Compare commits

...
7 Commits
Author SHA1 Message Date
yonlu b505959934 Merge remote-tracking branch 'origin/main' into wails-v3
Build & publish Arch package / arch-package (push) Successful in 2m32s
CI / check (push) Successful in 3m10s
CI / e2e (push) Canceled after 0s
Search index maintenance / maintain-index (push) Canceled after 1h31m25s
2026-08-17 11:38:49 -04:00
yonluandClaude Opus 5 409bfd5e89 test(download): wait for the work, not for the state that precedes it
TestManagerEndToEndAutoPick waits for StateComplete and then asserts
that staging was released and the library was rescanned. Those happen
*after* the state is recorded: manager.go sets StateComplete, then
satisfies the request, then releases staging, then scans. So waiting on
the state is not waiting on either assertion, and on a loaded machine
the worker is descheduled in between and the test reads the world one
step too early:

    manager_test.go:209: staging not released: 1 dirs remain
    manager_test.go:218: library scans = 0, want 1

It passed alone every time and failed three times under a full-suite
run, which is the signature of a test race rather than a broken
manager — nothing here is wrong except what the test chose to wait on.
It blocks pushes, since the pre-push hook is exactly the loaded run.

It polls for the side effects now, through the waitFor this package
already has and already uses for the same reason one file over
(service_test.go waits for a request to become satisfied after the same
StateComplete).

Not reproduced on demand: eight spinners and -count=5 did not provoke
it with or without the fix, so this rests on the ordering being plain
in the code rather than on a red-to-green demonstration.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AfVYUVExXsx1nSWrXN8mAh
2026-08-17 09:22:51 -04:00
yonluandClaude Opus 5 0eeef6048e feat(frontend): credit the artists on the full-screen now playing too
The phone shell's now-playing view landed on main while the credit
rendering was being written, so it arrived with the one call site that
still showed a multi-artist credit as a single link with the other
artists as punctuation inside it.

It is the same fix as the other ten: render from the parts, fall back
to the single link when there are fewer than two. The subscription is
what makes it show up at all — credits arrive after the track does, so
the name already on screen has to be re-rendered when they land.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AfVYUVExXsx1nSWrXN8mAh
2026-08-17 08:34:27 -04:00
yonlu 4fc0cdeab7 Merge remote-tracking branch 'origin/main' into wails-v3 2026-08-17 08:29:21 -04:00
yonluandClaude Opus 5 eb059a3d71 fix(database): retire a table whose shape the schema moved past
`applySchema` is CREATE ... IF NOT EXISTS and there is no migration
chain, so a *changed* table never migrates: the statement silently
no-ops against the old shape. Two plans had already landed on that, and
neither showed up in a test because a fresh install is perfectly
healthy.

- 014 added `total_tracks` to explore_index and to `indexRowFields`,
  the projection every explore read uses, so every search, browse,
  artist page and album page failed with "no such column: total_tracks"
  on any database that already had a catalog.
- 013 reshaped audio_files, so applySchema could not run at all and the
  app did not open.

staleshape.go runs before applySchema and drops what disagrees, so the
create is a create. It parses sql/schemas/ for the expectation rather
than writing the column list down a second time, and it notices a
changed *type* as well as a missing column — 013 moved mbid TEXT to
BLOB, which no ALTER could express and which SQLite will not coerce, so
a query against 16 raw bytes returns no rows rather than an error.

Only Authored tables are exempt. Cache is rebuildable by definition,
Owned is what a rescan rebuilds (plan 013's stated "delete and
rescan"), and a table the schema no longer describes at all goes too --
013 left seven behind plus schema_migrations.

Three things in it are load-bearing, and each was a bug first:

- The parser read `UNIQUE(mbid)` as a column, which made a healthy
  catalog look stale. That would have retired it on every launch and
  cost every user an artifact download per start.
- The drops are one transaction with defer_foreign_keys. Those legacy
  tables reference each other, so any order fails on whichever goes
  first; turning foreign keys off instead would suppress
  playlist_tracks.audio_file_id's ON DELETE SET NULL and leave entries
  pointing at ids a rescan reissues to *different songs*. Nulled
  entries are empty; stale ones are wrong, and wrong quietly.
- The order is sorted, so a failure reproduces. Map order is random,
  and the foreign-key bug passed its own regression test on two runs in
  three until the order was fixed.

Verified against a real pre-013 install: it opens, its 22 playlists
survive, 1,887 linked playlist entries become 0 rather than dangling,
and the legacy tables are swept.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AfVYUVExXsx1nSWrXN8mAh
2026-08-17 08:27:05 -04:00
yonluandClaude Opus 5 dcabec8b1d feat(frontend): render a multi-artist credit as one link per artist
Every artist name in the app went through `artistLink(name, mbid)`, so
a track credited to several artists rendered one link and the rest as
punctuation — "2Pac feat. Snoop Dogg" linked 2Pac and left Snoop Dogg
as text inside it.

`creditLink(parts, fallbackName, fallbackMbid)` renders the credit from
its parts: one link per credited artist, join phrases as plain text
between them. The link boundaries are known by construction, which is
the point — locating a name inside the stored credit string would
reintroduce the mismatch the catalog exists to avoid, since that string
may come from the file's tags while the parts come from MusicBrainz and
the two disagree for ~1 in 3 multi-artist credits.

Fewer than two parts falls through to the previous behaviour exactly,
so a single-artist credit, a file with no recording MBID and a catalog
that has not answered yet all render as they did before. Nothing tries
to split the fallback string: "Simon & Garfunkel" is one artist, which
is why primaryArtist() does not split on "&" either.

The lookup is keyed on the recording MBID, which both sides already
carry — a catalog row has one and so does a local file — so one binding
serves Explore and the library's own lists, and no local table is
needed for this.

credit-store.ts, and three things in it are load-bearing:

- A miss is cached as an empty array. The backend returns nothing for a
  single-artist credit, which is ~87% of tracks, and caching only the
  hits would re-request the rest on every render forever.
- request() is per-row and coalesces into one call per frame. A
  virtualized list cannot hand over "the whole list": 50,000 rows would
  be 100 queries for the ~30 on screen.
- It is an LRU with a counted retainedChars probe, because a cache that
  grows with use is a leak with a schedule.

The virtualized lists push requestUpdate() into the virtualizer rather
than only the host, since its rows come from its own properties — a
host update alone would leave them exactly as they were. now-playing
marks its geometry dirty instead, because the marquee measures the text
it is about to scroll.

track-list keeps the single link while a search term is active: the
highlight spans are computed against the flat credit string, and
mapping them onto decomposed parts is a different problem.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AfVYUVExXsx1nSWrXN8mAh
2026-08-17 08:26:34 -04:00
yonluandClaude Opus 5 b3737d30af feat(explore): carry multi-artist credits in the catalog
A track credited to more than one artist has exactly one navigable
artist in this app and the rest are punctuation. `primaryArtist()`
string-parses the credit, strips a " feat. " clause and discards the
guest; it deliberately does not split on "&", "with" or "," because
those live inside real artist names.

Measured on a real 26,069-file library plus an 80+80 MusicBrainz
sample: 13% of recordings are multi-artist upstream, while only 0.86%
of files carry any structured multi-artist tag — mp3 carries zero
files with multiple MUSICBRAINZ_ARTISTID across 19,840. Of 1,286 files
saying "feat.", 90% have nothing structured behind it, and a sample of
80 such files was multi-artist in MB 80 times out of 80.

CLAUDE.md justified plan 013's removal of the credit tables with "3
credits of 2,823 listed more than one artist". That measured our own
*writer* — cachedLinkArtist was called once per credit, so a
collaboration could never have been recorded. Dropping the join table
was still right on cost; the evidence for "multi-artist is rare" was
not.

A credit is ordered parts and the credit string is derived from them,
so join phrases are assembly instructions, not disassembly ones.
Nothing here reconstructs a credit by searching a name inside a credit
string: the stored text may come from tags while the parts come from
the catalog, and those disagree for ~1 in 3 multi-artist credits.

Where it comes from, after two dead ends: the canonical dump CI
already streams has no join phrases and no as-credited names, and the
JSON dumps cover 153,691 recordings of ~35M with *zero* overlap
against a real library. So mbdump.tar.bz2 — 7.1 GB, ~13.7 min in
pure-Go bzip2, whose members are alphabetical, which is what lets one
pass resolve an entity's credit without buffering 35M recordings.

- artist_credit_part / artist_credit_ref, multi-artist credits only:
  a single-artist credit is already explore_index's own artist_name.
- Column layouts verified against the real 20260815 export;
  ErrDumpShape makes a wrong guess a failed build, not a wrong catalog.
- The pass runs on every mode, not just a build. The job picks its mode
  from the index's own state, and a complete import means "refresh",
  which never enters the importer — so credits could otherwise only
  arrive via a rebuild that re-downloads ~205 GB. It reports whether it
  populated anything, which is what flips `changed` and republishes.
- The importer asks whether an artifact carries the tables, on the
  writer where `core` is attached, so the artifact already published
  still imports.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AfVYUVExXsx1nSWrXN8mAh
2026-08-17 08:25:36 -04:00
38 changed files with 4002 additions and 42 deletions
@@ -0,0 +1,337 @@
# 015 — Multi-artist credits, navigable
## The problem
A track credited to more than one artist has exactly one navigable
artist in this app, and the others are punctuation.
`audio_files` carries `artist_credit` (the credit as tagged, for
display) and `artist_id` (one artist, for grouping and browsing).
`primaryArtist()` (`backend/library/artistcredit.go:53`) resolves that
one artist by *string-parsing* the credit: it strips a " feat. "
clause, and deliberately does not split on `&`, `x`, `with` or `,`
because those appear inside real artist names. So "Lana Del Rey ft.
Sean Lennon" stores Lana Del Rey and discards Sean Lennon entirely,
and "Alina Baraz & Galimatias" stores one artist whose name is the
whole credit.
### What the measurement says
Measured 2026-08-16 against a real 26,069-file library (19,840 mp3,
6,229 flac; 57 unreadable, m4a/ogg not examined), plus an 80+80
MusicBrainz `inc=artist-credits` sample.
- **13%** of a random sample of the library's recordings have more
than one credited artist in MusicBrainz (10 of 79 resolved).
Extrapolates to ~3,250 of the 24,989 files carrying a recording
MBID.
- **0.86%** of files (224) carry any structured multi-artist signal in
their own tags. mp3 carries **zero** files with multiple
`MUSICBRAINZ_ARTISTID` values across 19,840 files; flac has 87.
- **1,286** files say "feat." in `ARTIST`; **1,159 of them (90%)**
have nothing structured behind it. A sample of 80 such files was
multi-artist in MB **80 of 80 times**.
CLAUDE.md currently justifies plan 013's removal of `artist_credit` /
`artist_credit_artist` with "3 credits of 2,823 listed more than one
artist". That figure measured **our own writer**, not the library:
`cachedLinkArtist` was called exactly once per credit
(`e7748f1^:backend/library/library.go:1842`), so a collaboration could
never have been recorded, and the three were resolution collisions on
shared credit text. Dropping the join table was still correct — it only
ever held one row, so it was pure join cost — but the stated evidence
does not support "multi-artist is rare". Correcting that claim is part
of this plan.
### Why the tags cannot answer it
Deriving the decomposition locally, with no network, works **79% of the
time** (169 of 215 files with a multi-value `ARTISTS` tag: mp3 69/105,
flac 100/110), and the failures are systematic rather than random:
```
ARTIST = '2Pac feat. Snoop Dogg, Nate Dogg, Hussein Fatal & Yaki Kadafi'
ARTISTS = ['2Pac', 'Snoop Doggy Dogg', 'Nate Dogg', 'Fatal', 'Yaki Kadafi']
```
`ARTISTS` holds **canonical** artist names; `ARTIST` holds
**as-credited** names. Locating one inside the other fails on
"Snoop Doggy Dogg" vs "Snoop Dogg", on "Fatal" vs "Hussein Fatal", and
on Unicode (`Michel'le` vs `Michelle`, `K-Ci` vs `KCi` — U+2010, not
a hyphen). That distinction is precisely what a join phrase encodes,
and it is why this cannot be a tag-parsing feature.
Two format details that will mislead anyone re-running the probe:
Picard writes `ARTISTS` **slash-joined into one TXXX frame** on mp3 and
as **true repeated Vorbis keys** on flac, so a probe splitting only on
NUL undercounts mp3 to zero.
## The shape
MusicBrainz models a credit as ordered parts, and the credit *string*
is derived from them — `artist_credit.name` is a cached render, nothing
more. Each participant is `(position, artist, name, join_phrase)`,
where `artist` is the MBID (canonical, what you navigate to) and `name`
is the credited spelling (what you display).
**Join phrases are assembly instructions, not disassembly
instructions.** Rendering is a concatenation, never a search:
```
for each (position, artist_mbid, credited_name, join_phrase):
emit link(credited_name -> artist_mbid)
emit text(join_phrase)
```
The link positions are known **by construction**. This is load-bearing:
if we instead located each `credited_name` inside the stored
`artist_credit` text, we would reintroduce the mismatch above — the
stored string may have come from the tags while the parts come from the
catalog, and those **disagree for ~1 in 3 multi-artist files** (61 of
90 sampled credits rendered exactly equal to the tag string).
Divergences seen: `'Skrillex feat. Swae Lee'` tagged vs
`'Skrillex & Swae Lee'` in MB; `'STRFKR'` vs `'Starfucker'`;
`'Zedd feat. Hayley Williams'` vs `'... of Paramore'`. Either MB was
edited after tagging or Picard versions differ; either way the search
would miss or match the wrong span.
So `audio_files.artist_credit` stops being the source of truth and
becomes the **fallback**, used only where there are no parts.
## Where the data comes from
The catalog carries the decomposition; no user ever makes a
per-recording call. Two sources were ruled out first, both cheaply:
- **The canonical dump — which is what CI already pulls
(`dumpimport.go:84-85`) — does not have it.**
`canonical_musicbrainz_data.csv` gives `artist_mbids` (ordered list)
and `artist_credit_name`, but that last column is the *rendered*
string. Splitting it on CI needs the as-credited names, so CI would
fail exactly the way a local parse does.
- **The JSON dumps do not cover the catalog.**
`json-dumps/recording.tar.xz` is 31 MB / 368 MB uncompressed and
holds **153,691 recordings**, not ~35M. Measured against the test
library's 24,885 recording MBIDs: **0.00% overlap, zero rows**. It is
some other subset and is not usable.
That leaves the core dump, **`mbdump.tar.bz2`** (7.1 GB compressed at
the 20260815 export), from
`https://data.metabrainz.org/pub/musicbrainz/data/fullexport/`. Four
members are needed:
| member | why | approx rows |
| --- | --- | --- |
| `mbdump/artist_credit_name` | `(artist_credit, position, artist, name, join_phrase)` — the payload | ~4M |
| `mbdump/artist` | `id -> gid`, since the above references artist *row ids* | ~2.6M |
| `mbdump/recording` | `gid -> artist_credit`, to key credits by recording MBID | ~35M |
| `mbdump/release_group` | same, for album credits | ~2M |
### Coverage is not a concern
Of 24,885 distinct recording MBIDs in the test library, **24,808
(99.7%)** already have an `explore_index` recording row, measured
against a database at 2,052,200 rows — i.e. shipped-artifact coverage,
not a local build's. The popularity filter does not strand the long
tail here.
## Status
- **Phase 1 — done.** `backend/explore/dumpcredits.go` +
`dumpcreditswrite.go`, wired into `dumpimport.go`'s `run` behind its
own `credits_import_done` marker.
- **Phase 2 — done.** `cmd/indexexport` writes the two tables;
`artifactimport.go` reads them behind `artifactHasCredits()`.
- **Phase 4 — done, and it does not need Phase 3.** `explore.GetCredits`
reads the catalog tables keyed on the *recording* MBID, which both
sides of the app already carry — a catalog row has one and so does a
local file (`library.Track.RecordingMBID`). So one binding serves the
Explore pages and the library's own lists, and all ten artist-link
call sites render credits today without a local table.
- **Phase 3 (`file_artists`) — not started, and now an
offline-resilience task rather than a prerequisite.** The table is
deliberately *not* declared yet: nothing writes or reads it, and a
schema file plus a datamap note describing behaviour that does not
exist is a claim the code cannot back. Its remaining
value is that credits currently vanish when the catalog is absent or
still downloading, which is precisely the `no-index` state
`ShelfPage.State` exists to describe. Materialising into
`file_artists` is what makes a library stand on its own.
**Nothing renders yet in practice**, because no published artifact
carries credit tables — every credit falls back to its single link
until an index build with Phase 1 runs and is exported.
**Column layouts are verified against the real 20260815 export**, not
taken from the schema docs — `artist(id, gid, …)`,
`artist_credit(id, name, artist_count, …)`,
`artist_credit_name(credit, position, artist, name, join_phrase)` and
`recording(id, gid, name, artist_credit, …)` were each read out of the
dump. `release_group` shares `recording`'s first four columns and is
the one layout still taken on trust; `ErrDumpShape` turns a wrong guess
into a loud failure rather than a quietly wrong catalog.
**Still unrun: the ingest against the real 7.1 GB dump.** Everything is
covered by tests over a synthetic tar, which cannot catch a surprise in
the other ~35M rows.
### Phase 1 — Ingest credits on CI
New dump stage in `cmd/indexbuild`, behind the `indexbuild` tag with
the rest of `dumpimport.go`'s stages.
**Constraint from `b98840e`:** `cmd/indexbuild` is built
`CGO_ENABLED=0` in a plain `golang` container and must not reach the
Wails `application` package — `TestIndexToolsDoNotImportWails` walks
`go list -deps -tags indexbuild`. Nothing here should need it, but a
new `ServiceStartup` hook on a package this imports is how it comes
back. Go's `compress/bzip2` is pure Go and decompress-only, which is
all this needs.
**Measured, 20260815 export.** Tar members are **alphabetical**, and
that is favourable: `artist` (435 MB), `artist_credit` (414 MB) and
`artist_credit_name` (237 MB) all fall inside the first ~900 MB
compressed, while `recording` and `release_group` come later. So the
maps are complete before the rows that consume them arrive, and no
recording data is ever buffered.
Pure-Go `compress/bzip2` decompresses at **26 MB/s uncompressed /
8.7 MB/s compressed** (measured on a 250 MB prefix, 3.01x ratio) —
**~13.7 min** for the whole file single-threaded, and less because the
stream can stop after `release_group` rather than reading the
`series`/`tag`/`track`/`url`/`work` tail. The 2 MB/s origin throttle
dominates, as it already does for every other dump here.
Do not, however, *depend* on the ordering: assert it and fall back to
buffering if a future export reorders, rather than silently emitting
nothing.
- `artist` -> `map[int32]uuid16` (~2.6M x ~20 B = ~60 MB)
- `artist_credit_name` -> `map[int32][]creditPart` (~4M x ~40 B =
~200 MB)
- `recording` / `release_group` -> emit `gid -> credit_id` **only for
MBIDs already in `explore_index`** (the kept set is ~1.4M x 16 B =
~22 MB), which is what keeps 35M rows from being held
Peak ~300 MB, one sequential pass.
**Only multi-artist credits are stored.** A single-artist credit is
`(name, "")` and is already fully described by `explore_index`'s
`artist_name` / `artist_mbid`; storing it would triple the table for
nothing. Post-filter after loading, once the row count per credit is
known.
New tables (and `datamap` entries, or `TestCatalogCoversSchema` fails
the build — both are `Cache`, matching `explore_index`):
```
artist_credit_part(credit_id, position, artist_mbid, credited_name, join_phrase)
```
with `explore_index.artist_credit_id` as the link. Credits are
**shared** — an album's twelve tracks by one artist share one credit
row — which is the opposite of 013's local verdict, and correctly so:
1:1 in a local library, genuinely many-to-one at 2M-row catalog scale.
### Phase 2 — Ship them in the artifact
`cmd/indexexport` currently creates exactly two tables in the artifact
(`explore_index`, `artifact_meta`, at `cmd/indexexport/*.go:147,170`),
so this is a structural addition, not a column.
Estimated size: ~13% of 1.4M recordings, deduplicated by shared credit,
at ~2.3 parts each — order 400k rows, ~18 MB uncompressed. Against a
~0.6 GB install that is acceptable; it must be measured rather than
assumed before merge.
`artifactimport.go` must read it **only if present**, on the writer
handle where `core` is attached — the `artifactHasTotals()` /
`artifactStoresText()` pattern (`artifactimport.go:145-175`), one step
up from a column to a table. An artifact published before this exists
is still a perfectly good catalog and must import as one that declines
to answer. Adding this to the importer's SELECT list without the probe
is how every already-published artifact starts failing.
`artifactCatalogColumns` gains `artist_credit_id`; it is kept in sync
with the exporter by `TestArtifactColumnsMatchExporter`.
### Phase 3 — Materialize locally
```
file_artists(audio_file_id, position, artist_id, credited_name, join_phrase)
```
`credited_name` is stored **per row**, not looked up from
`artists.name` — that is the Snoop-Doggy-Dogg distinction, and it is
the whole point.
Filled at scan/import time by joining `audio_files.recording_mbid`
against the catalog. **Materialized rather than resolved live**,
because the catalog is a downloaded artifact that can be absent or
still arriving — that is why `ShelfPage.State` has a `no-index` value —
and a library whose track rows lose their artists when the catalog is
missing is worse than today.
That implies a backfill for the case where the catalog arrives *after*
the library was scanned. It registers with `jobs` (progress, cancel)
like every other long pass, and takes a **distinct kind** from
`index-build`, since `job-controls.ts` keys its "you will discard hours
of downloading" confirmation on that kind.
`artists` gains rows for guests who own no files. **This changes what
the artists grid shows** and is an open question below.
### Phase 4 — Render
`utils/explore-link.ts` gains a credit-rendering entry point taking
ordered parts and returning a `TemplateResult`. Every row and detail
view already renders artist names through it, so they inherit
multi-artist links without individually knowing credits exist — the
property that made centralising it worthwhile.
Its existing fallback philosophy already covers the no-parts case: "a
list where some rows are clickable and others silently are not reads as
a bug, not as a statement about metadata." Where there are no parts
(no recording MBID, or no catalog row — ~4% of the test library) render
today's behaviour: the flat `artist_credit` string with one link to the
primary artist. **Do not split the string there.** There is genuinely
no information to split on, and that is the one place the temptation
returns.
`primaryArtist()` stays exactly as it is. It remains the fallback and
is still what `artist_id` means.
## Open questions
1. **Catalog credit vs tagged credit, when they disagree** (~1 in 3
multi-artist files). Rendering the catalog's decomposition is what
makes names navigable; preserving the file's is what makes the app
reflect the user's files. Leaning toward: render the catalog
decomposition, keep `artist_credit` as the fallback string. Wants a
deliberate decision, not an accident.
2. **Do guest artists appear in the artists grid?** Phase 3 creates
`artists` rows for people who own no files. The grid currently means
"artists in your library" and joins `audio_files`. A guest on one
track is arguably in the library and arguably not. Whichever way,
the ownership question stays "is there a file" — that rule does not
bend.
3. **`release_group` credits** are ingested in the same pass for
nearly nothing, but album-artist rendering is a separate surface.
Ship the data in phase 1, render in a follow-up rather than widening
phase 4.
4. **Our own `tagwriter`** does not write `ARTISTS` or multiple
`MUSICBRAINZ_ARTISTID` frames, so autotagging a folder degrades the
very field this rests on — the same shape as the existing
track-totals note. Out of scope here; worth recording.
## Verification
- Coverage: re-run the library probe and assert `file_artists` is
populated for ~13% of files, not ~0.9%.
- `TestCatalogCoversSchema` / `TestLifetimesMatchSchema` for the new
tables.
- `TestIndexToolsDoNotImportWails` still passes with the new stage.
- An artifact **without** the credits table imports cleanly (the
`artifactHasTotals` regression shape).
- Round-trip: a known multi-artist recording renders each name as a
separate link with the correct join phrases between them.
+45
View File
@@ -233,6 +233,51 @@ rather than renaming them.
the drift it caused before — `sql/schemas/` and the migrations
disagreed, and sqlc generated against the stale one.
**What that costs an existing database is repaired once, at open.**
`CREATE ... IF NOT EXISTS` reaches an existing table only if its shape
already matches and otherwise silently no-ops, so a *changed* table
never migrates. Plan 014 added `total_tracks` to `explore_index` and
to `indexRowFields` — the projection every explore read uses — and no
database that already existed grew the column: **every** Explore
search, browse, artist and album page on such an install failed with
`no such column: total_tracks`, while a fresh install was perfectly
healthy, which is exactly why no test saw it. Plan 013 was worse on
the same install: `applySchema` could not be applied at all over a
pre-013 `audio_files`, so the app did not open.
`backend/database/staleshape.go` runs before `applySchema` and
retires what is stale, so the create is a create. Five things about
it are load-bearing:
- **It parses `sql/schemas/` for the expectation** rather than
writing the column list down a second time, because a second list
is a second thing to forget — the fault it exists to repair.
- **It notices a changed *type*, not just a missing column.** 013
moved `mbid` from TEXT to BLOB, and SQLite does not coerce between
them: a comparison against 16 raw bytes returns no rows rather than
an error. `ALTER TABLE ADD COLUMN` would have handled
`total_tracks` alone and cannot express this at all, which is why
the repair drops rather than migrates.
- **`Authored` is never retired**, and that boundary is a test
(`TestAuthoredTablesAreNeverRetired`), not a comment. Everything
else is rebuildable: `Cache` by definition, `Owned` by a rescan —
plan 013's stated "delete and rescan" — and `Derived` from Owned.
A table the schema no longer describes at all goes too; 013 left
seven behind plus `schema_migrations`.
- **The drops are one transaction with `defer_foreign_keys`.** Those
legacy tables reference each other, so dropping them in any order
fails on whichever goes first, and turning foreign keys *off*
instead would silently take `playlist_tracks.audio_file_id`'s
ON DELETE SET NULL with it — leaving playlist entries pointing at
ids a rescan reissues to *different songs*. Nulled entries are
empty; stale ones are wrong, and wrong quietly.
- **The order is sorted, so a failure reproduces.** Map order is
random, and the foreign-key bug above passed its own regression
test on two runs in three until the order was fixed.
Retiring `explore_index` takes its FTS and its meta with it, because
the `dump_import_done` marker is what would otherwise stop the
artifact ever being fetched again.
**What that costs an existing database is that it does not open**, and
"delete and rescan" is the answer (plan 013, open question 1) — free
for everyone except one machine. The index job's `/cache` volume is a
+8
View File
@@ -89,6 +89,14 @@ func NewDB(logger *slog.Logger) (*DB, error) {
return nil, fmt.Errorf("could not apply PRAGMAs: %w", err)
}
// Before the schema is applied, not after: applySchema is
// CREATE ... IF NOT EXISTS, which no-ops against a table that
// already exists in an older shape. Retiring the stale one first is
// what turns that no-op into a create.
if err := retireStaleTables(dbCtx, db, logger); err != nil {
return nil, err
}
if err := applySchema(dbCtx, db); err != nil {
return nil, err
}
@@ -0,0 +1,56 @@
-- The decomposition of a multi-artist credit, from the MusicBrainz
-- dump. One row per credited artist, in credit order.
--
-- A credit is ordered parts, and the credit *string* is derived from
-- them -- MusicBrainz's own `artist_credit.name` is a cached render and
-- nothing more. Rendering is a concatenation:
--
-- for each part in position order:
-- emit link(credited_name -> artist_mbid)
-- emit text(join_phrase)
--
-- so the link boundaries are known by construction. That is the whole
-- reason this table exists, and it is why nothing may reconstruct a
-- credit by *searching* for a name inside a credit string: the stored
-- string may have come from a file's tags while the parts come from the
-- catalog, and measured on a real library those disagree for about one
-- in three multi-artist credits ("Skrillex feat. Swae Lee" tagged
-- against "Skrillex & Swae Lee" upstream). A search would miss, or
-- match the wrong span.
--
-- `credited_name` is the name *as credited*, which is not the artist's
-- canonical name: MusicBrainz credits "Snoop Dogg" on a track by the
-- artist whose name is "Snoop Doggy Dogg". It is stored per row rather
-- than joined from an artist table for exactly that reason.
--
-- Only *multi-artist* credits are stored. A single-artist credit is
-- (name, "") and is already fully described by explore_index's
-- artist_name and artist_mbid; storing those would roughly triple the
-- table to say nothing new.
--
-- Credits are shared: an album's twelve tracks by one artist reference
-- one credit_id. That is the opposite of the local library's verdict
-- in plan 013, and correctly so -- credit sharing is 1:1 in one
-- person's files and genuinely many-to-one across a 2M-row catalog.
--
-- MBIDs are the same 16 raw bytes explore_index stores, for the same
-- size reason and with the same CHECK, so a stringly write fails at the
-- insert that made it rather than reading back as no rows at all. See
-- backend/explore/mbid.go.
CREATE TABLE IF NOT EXISTS artist_credit_part (
credit_id INTEGER NOT NULL,
position INTEGER NOT NULL,
artist_mbid BLOB NOT NULL CHECK(length(artist_mbid) = 16),
-- The name as credited on this release, which may differ from the
-- artist's canonical name. Display uses this; navigation uses the
-- MBID above.
credited_name TEXT NOT NULL,
-- The literal connector that follows this part -- " feat. ", " & ",
-- ", ", or "" on the last part. Rendered as plain text between two
-- links.
join_phrase TEXT NOT NULL DEFAULT '',
PRIMARY KEY (credit_id, position)
) WITHOUT ROWID;
@@ -0,0 +1,30 @@
-- Which credit a catalog entity is credited to. One row per recording
-- or release group whose credit names more than one artist.
--
-- This is a table rather than an `explore_index.artist_credit_id`
-- column, and that is a deliberate consequence of how this app applies
-- its schema. `applySchema` is CREATE ... IF NOT EXISTS and there is
-- no migration chain (plan 013), so a *column* added to an existing
-- table never reaches a database that already has it -- while a new
-- *table* is created on every install, old or new, for free.
-- explore_index is the one table nobody can afford to drop and rebuild
-- on a schema change: it is the artifact users download rather than
-- derive.
--
-- Only multi-artist credits are referenced here, matching
-- artist_credit_part. An entity with no row is credited to exactly one
-- artist, which explore_index's own artist_name and artist_mbid already
-- describe -- so absence is the common case and means "nothing to
-- decompose", not "unknown".
--
-- `credit_id` is opaque and is only meaningful against the
-- artist_credit_part rows built or imported alongside it. The two are
-- always written together; nothing persists a credit_id anywhere else.
-- The local library stores resolved parts, never this id.
CREATE TABLE IF NOT EXISTS artist_credit_ref (
mbid BLOB NOT NULL PRIMARY KEY CHECK(length(mbid) = 16),
credit_id INTEGER NOT NULL
) WITHOUT ROWID;
CREATE INDEX IF NOT EXISTS idx_artist_credit_ref_credit
ON artist_credit_ref(credit_id);
+13
View File
@@ -27,6 +27,19 @@ type Artist struct {
Mbid sql.NullString
}
type ArtistCreditPart struct {
CreditID int64
Position int64
ArtistMbid []byte
CreditedName string
JoinPhrase string
}
type ArtistCreditRef struct {
Mbid []byte
CreditID int64
}
type ArtistEnrichment struct {
ArtistMbid string
BrowsedAt sql.NullTime
+552
View File
@@ -0,0 +1,552 @@
package database
import (
"context"
"database/sql"
"fmt"
"io/fs"
"log/slog"
"maps"
"path"
"slices"
"strings"
"yellowjacket/backend/datamap"
)
// This file repairs the one thing `CREATE TABLE IF NOT EXISTS` cannot.
//
// `sql/schemas/` is the single description of the schema and there is no
// migration chain (plan 013): a schema change is one edit to one file.
// That works perfectly for a *new* table, which every install then
// creates, and not at all for a changed one -- `IF NOT EXISTS` reaches
// an existing table only if its shape already matches, and otherwise
// silently no-ops. The user's answer to that is "delete and rescan"
// (plan 013, open question 1), which is free for everything a rescan
// rebuilds.
//
// It is not free for the catalog. explore_index is a *downloaded
// artifact*, not something derived from the user's files, and it is the
// largest thing this app stores. So it went stale instead: plan 014
// added `total_tracks` to the schema and to `indexRowFields` -- the one
// projection every explore read uses -- and no database that already
// existed ever grew the column. Every Explore search, browse, artist
// page and album page on such an install fails with
// "no such column: total_tracks", while a fresh install is perfectly
// healthy, which is why the tests did not see it. The same databases
// are stale a second way, from the same plan: their `mbid` columns are
// still TEXT where the schema now declares BLOB, and SQLite does not
// coerce between the two -- a comparison against 16 raw bytes simply
// returns no rows.
//
// The repair is to notice and drop, not to migrate. A dropped catalog
// costs one artifact download (about a minute); the alternative --
// ALTER TABLE ADD COLUMN, which would handle `total_tracks` alone
// cheaply -- cannot express the TEXT-to-BLOB half at all, and would
// leave those installs quietly broken while reporting success.
//
// Everything except `Authored` is eligible. `Cache` is rebuildable by
// definition; `Owned` is a projection of the user's files and a rescan
// rebuilds it, which is plan 013's stated answer to exactly this
// situation ("delete and rescan", open question 1); `Derived` is
// computed from Owned. No `Authored` table is ever dropped here --
// that is the whole point of the datamap, and it is asserted by
// TestAuthoredTablesAreNeverRetired rather than only stated.
//
// What that does *not* buy is immunity for authored rows that reference
// a retired table. `audio_files` is MIXED KIND: `play_count`,
// `last_played` and `tag_status` are authored columns on an Owned
// table, and they go with it. Playlists survive as playlists, and
// their entries survive pointing at nothing. That cost was weighed and
// accepted rather than overlooked -- the alternative is to carry the
// authored columns across the rebuild keyed on file_path, which stays a
// real option if this ever bites harder than it is worth.
//
// **This relies on foreign_keys being ON**, which applyPRAGMAs has
// already done by the time NewDB calls it, and the dependency is not
// cosmetic. SQLite performs an implicit DELETE before dropping a table
// when foreign keys are enabled, so `playlist_tracks.audio_file_id` --
// declared ON DELETE SET NULL -- is nulled. With foreign keys off, no
// action fires and those rows keep the ids they had, which a rescan
// then reissues starting from 1: every playlist would silently fill
// with *different songs*. Nulled entries are merely empty; stale ones
// are wrong, and wrong quietly. TestRetiringOwnedTablesDoesNotDangle
// is what stops a future reordering turning one into the other.
// retireGroups are tables that must be retired together. A catalog
// whose rows are gone must not keep the full-text index built over
// them, nor the metadata claiming the import that produced them
// finished -- that marker is exactly what stops the artifact being
// fetched again. applySchema recreates all three empty immediately
// afterwards, and the ordinary "no index yet" path takes over.
var retireGroups = [][]string{
{
"explore_index",
"explore_index_fts",
"explore_index_meta",
"explore_champion_fts",
},
}
// schemaColumn is one column as the schema file declares it.
type schemaColumn struct {
name string
typ string
}
// retireStaleTables drops every non-authored table whose live shape no
// longer matches what sql/schemas/ declares, plus any table the schema
// no longer describes at all, so applySchema can create the current
// shape afresh. It runs before applySchema and is a no-op on a new
// database, where the tables do not exist yet.
func retireStaleTables(
ctx context.Context, db *sql.DB, logger *slog.Logger,
) error {
declared, err := declaredTables()
if err != nil {
return err
}
stale := make(map[string]string)
for table, columns := range declared {
entry, ok := datamap.Lookup(table)
if !ok || entry.Kind == datamap.Authored || entry.FTS {
continue
}
reason, err := staleReason(ctx, db, table, columns)
if err != nil {
return err
}
if reason != "" {
stale[table] = reason
}
}
obsolete, err := obsoleteTables(ctx, db)
if err != nil {
return err
}
maps.Copy(stale, obsolete)
if len(stale) == 0 {
return nil
}
return retireGroupsFor(ctx, db, logger, stale)
}
// obsoleteTables are live tables the schema no longer describes at all.
// TestCatalogCoversSchema makes the datamap a complete description of
// the current schema, so a table it does not know is one a past version
// created and this one does not -- plan 013 alone left seven behind
// (recordings, release_groups, artist_credit, artist_credit_artist,
// release_group_recordings, recording_genres) plus the
// schema_migrations table that squashing the chain retired. They are
// dead weight, and one of them holding a foreign key into a table being
// rebuilt is worse than dead weight.
//
// SQLite's own bookkeeping and FTS shadow tables are not obsolete:
// datamap.Lookup resolves a shadow table to its parent, and IsInternal
// covers the rest.
func obsoleteTables(ctx context.Context, db *sql.DB) (map[string]string, error) {
rows, err := db.QueryContext(
ctx, "SELECT name FROM sqlite_master WHERE type = 'table'",
)
if err != nil {
return nil, fmt.Errorf("could not list tables: %w", err)
}
defer func() { _ = rows.Close() }()
out := make(map[string]string)
for rows.Next() {
var name string
if err := rows.Scan(&name); err != nil {
return nil, fmt.Errorf("could not scan table name: %w", err)
}
if datamap.IsInternal(name) {
continue
}
if _, known := datamap.Lookup(name); !known {
out[name] = "the schema no longer describes this table"
}
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("could not read table list: %w", err)
}
return out, nil
}
// retireGroupsFor drops each stale table along with everything its
// retire group says must go with it.
func retireGroupsFor(
ctx context.Context, db *sql.DB, logger *slog.Logger,
stale map[string]string,
) error {
drop := make(map[string]string)
for table, reason := range stale {
drop[table] = reason
for _, group := range retireGroups {
if !slices.Contains(group, table) {
continue
}
for _, member := range group {
if _, already := drop[member]; !already {
drop[member] = "retired with " + table
}
}
}
}
return dropDeferred(ctx, db, logger, drop)
}
// dropDeferred drops every named table in one transaction with foreign
// key enforcement deferred to the commit.
//
// The deferral is required and the two obvious alternatives are both
// wrong. These tables reference each other -- pre-013 `audio_files`
// has a foreign key into `recordings`, which is itself being retired --
// so dropping them one at a time in an arbitrary order fails with
// "FOREIGN KEY constraint failed" on whichever is unlucky enough to go
// first, and there is no order that is safe in general. Turning
// foreign keys *off* for the duration would fix that and silently take
// the ON DELETE SET NULL on `playlist_tracks.audio_file_id` with it,
// leaving playlist entries pointing at ids a rescan reissues to
// different songs -- the exact failure
// TestRetiringOwnedTablesDoesNotDangle exists to prevent.
//
// Deferring keeps the actions firing while tolerating the inconsistency
// in the middle, and the commit then checks that the end state is
// sound. It is set inside the transaction because SQLite resets it at
// every commit.
func dropDeferred(
ctx context.Context, db *sql.DB, logger *slog.Logger,
drop map[string]string,
) error {
tx, err := db.BeginTx(ctx, nil)
if err != nil {
return fmt.Errorf("could not begin the retire transaction: %w", err)
}
defer func() { _ = tx.Rollback() }()
if _, err := tx.ExecContext(ctx, "PRAGMA defer_foreign_keys = ON"); err != nil {
return fmt.Errorf("could not defer foreign keys: %w", err)
}
// Sorted, so a failure is reproducible. Map order is random, and a
// bug that depends on which table happens to go first reproduces on
// one run in three and passes review on the other two -- which is
// exactly how the foreign-key ordering above reached a real
// database. Sorting does not make any order *safe*; the deferral
// does that.
for _, table := range slices.Sorted(maps.Keys(drop)) {
logger.Warn(
"retiring a table the schema no longer describes",
"table", table,
"reason", drop[table],
)
if _, err := tx.ExecContext(
ctx, "DROP TABLE IF EXISTS "+quoteIdent(table),
); err != nil {
return fmt.Errorf("could not retire stale table %s: %w", table, err)
}
}
if err := tx.Commit(); err != nil {
return fmt.Errorf("could not commit the retire: %w", err)
}
return nil
}
// staleReason reports why a live table disagrees with its declaration,
// or "" when it agrees. A column the live table does not have is the
// additive case; a column whose declared type changed is the one an
// ALTER could not fix anyway. Columns the live table has and the
// schema no longer declares are ignored: they cost nothing and dropping
// the table over one would retire a healthy catalog.
func staleReason(
ctx context.Context, db *sql.DB, table string, columns []schemaColumn,
) (string, error) {
live, err := liveColumns(ctx, db, table)
if err != nil {
return "", err
}
if len(live) == 0 {
// Not present at all: applySchema is about to create it.
return "", nil
}
for _, col := range columns {
liveType, present := live[col.name]
if !present {
return "missing column " + col.name, nil
}
if !sameDeclaredType(col.typ, liveType) {
return fmt.Sprintf(
"column %s is %s, schema declares %s",
col.name, liveType, col.typ,
), nil
}
}
return "", nil
}
// liveColumns returns the live table's columns and their declared types,
// empty when the table does not exist.
func liveColumns(
ctx context.Context, db *sql.DB, table string,
) (map[string]string, error) {
rows, err := db.QueryContext(
ctx, "SELECT name, type FROM pragma_table_info(?)", table,
)
if err != nil {
return nil, fmt.Errorf("could not inspect table %s: %w", table, err)
}
defer func() { _ = rows.Close() }()
out := make(map[string]string)
for rows.Next() {
var name, typ string
if err := rows.Scan(&name, &typ); err != nil {
return nil, fmt.Errorf("could not scan column of %s: %w", table, err)
}
out[name] = typ
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("could not read columns of %s: %w", table, err)
}
return out, nil
}
// sameDeclaredType compares two SQLite type names. They are compared
// case-insensitively and only on the leading word, so INTEGER matches
// INTEGER and VARCHAR(20) matches VARCHAR -- SQLite's affinity rules
// make finer distinctions meaningless, and a difference that fine is
// not worth retiring a catalog over. An empty declared type matches
// anything, which is what a column declared with only constraints has.
func sameDeclaredType(declared, live string) bool {
d := strings.ToUpper(strings.Fields(declared + " ")[0])
l := strings.ToUpper(strings.Fields(live + " ")[0])
if d == "" || l == "" {
return true
}
if i := strings.IndexByte(d, '('); i >= 0 {
d = d[:i]
}
if i := strings.IndexByte(l, '('); i >= 0 {
l = l[:i]
}
return d == l
}
// declaredTables parses every CREATE TABLE in sql/schemas/ into its
// column list. Parsing the schema rather than writing the expectation
// down a second time is the point: a second list is a second thing to
// forget, which is the fault this whole file exists to repair.
func declaredTables() (map[string][]schemaColumn, error) {
dirEntries, err := schemas.ReadDir("sql/schemas")
if err != nil {
return nil, fmt.Errorf("could not read schemas directory: %w", err)
}
out := make(map[string][]schemaColumn)
for _, dirEntry := range dirEntries {
if dirEntry.IsDir() {
continue
}
content, err := fs.ReadFile(schemas, path.Join("sql/schemas", dirEntry.Name()))
if err != nil {
return nil, fmt.Errorf("could not read %s: %w", dirEntry.Name(), err)
}
maps.Copy(out, parseCreateTables(string(content)))
}
return out, nil
}
// constraintKeywords begin a table constraint rather than a column.
var constraintKeywords = map[string]bool{
"PRIMARY": true, "FOREIGN": true, "UNIQUE": true,
"CHECK": true, "CONSTRAINT": true,
}
// parseCreateTables extracts the column names and declared types of
// every non-virtual CREATE TABLE in one schema file.
func parseCreateTables(content string) map[string][]schemaColumn {
out := make(map[string][]schemaColumn)
rest := stripLineComments(content)
for {
idx := indexFold(rest, "CREATE TABLE ")
if idx < 0 {
return out
}
rest = rest[idx+len("CREATE TABLE "):]
head, body, ok := splitTableBody(rest)
if !ok {
return out
}
if name := tableName(head); name != "" {
out[name] = parseColumns(body)
}
}
}
// tableName pulls the table name out of the text between "CREATE TABLE"
// and its opening parenthesis, dropping an IF NOT EXISTS and any
// quoting.
func tableName(head string) string {
head = strings.TrimSpace(head)
head = strings.TrimPrefix(head, "IF NOT EXISTS ")
head = strings.TrimPrefix(head, "if not exists ")
fields := strings.Fields(head)
if len(fields) == 0 {
return ""
}
return strings.Trim(fields[len(fields)-1], `"'`+"`")
}
// splitTableBody returns the text before the table's opening paren and
// the balanced text inside it.
func splitTableBody(s string) (head, body string, ok bool) {
open := strings.IndexByte(s, '(')
if open < 0 {
return "", "", false
}
depth := 0
for i := open; i < len(s); i++ {
switch s[i] {
case '(':
depth++
case ')':
depth--
if depth == 0 {
return s[:open], s[open+1 : i], true
}
}
}
return "", "", false
}
// parseColumns splits a table body on its top-level commas and keeps
// the parts that are columns rather than table constraints.
func parseColumns(body string) []schemaColumn {
var (
out []schemaColumn
depth int
start int
)
parts := make([]string, 0, 8)
for i := range len(body) {
switch body[i] {
case '(':
depth++
case ')':
depth--
case ',':
if depth == 0 {
parts = append(parts, body[start:i])
start = i + 1
}
}
}
parts = append(parts, body[start:])
for _, part := range parts {
fields := strings.Fields(part)
if len(fields) == 0 {
continue
}
// A table constraint need not be followed by a space --
// "UNIQUE(mbid)" is one field, and reading it as a column name
// makes an entirely healthy table look stale, which retires a
// catalog nobody asked to lose.
head := fields[0]
if i := strings.IndexByte(head, '('); i >= 0 {
head = head[:i]
}
if constraintKeywords[strings.ToUpper(head)] {
continue
}
col := schemaColumn{name: strings.Trim(head, `"'`+"`")}
if len(fields) > 1 {
col.typ = fields[1]
}
out = append(out, col)
}
return out
}
// stripLineComments removes -- comments, which otherwise contribute
// stray parentheses and commas to the parse.
func stripLineComments(s string) string {
lines := strings.Split(s, "\n")
for i, line := range lines {
if idx := strings.Index(line, "--"); idx >= 0 {
lines[i] = line[:idx]
}
}
return strings.Join(lines, "\n")
}
// indexFold is a case-insensitive strings.Index.
func indexFold(s, substr string) int {
return strings.Index(strings.ToUpper(s), strings.ToUpper(substr))
}
// quoteIdent quotes a table name for interpolation into DDL, which
// cannot take a bound parameter.
func quoteIdent(name string) string {
return `"` + strings.ReplaceAll(name, `"`, `""`) + `"`
}
+499
View File
@@ -0,0 +1,499 @@
package database
import (
"context"
"database/sql"
"log/slog"
"path"
"testing"
_ "modernc.org/sqlite"
)
// testLogger discards the repair's warnings; the tests assert on the
// database, not on the log.
func testLogger() *slog.Logger {
return slog.New(slog.DiscardHandler)
}
// openRaw opens a scratch database file with no schema applied, so a
// test can build an *old* shape and then let NewDB's repair meet it.
func openRaw(t *testing.T, dir string) *sql.DB {
t.Helper()
db, err := sql.Open("sqlite", path.Join(dir, "yj.db"))
if err != nil {
t.Fatalf("open: %v", err)
}
t.Cleanup(func() { _ = db.Close() })
return db
}
// TestRetiresIndexMissingAColumn is plan 014's bug, symptom first: an
// explore_index created before `total_tracks` existed, met by the
// projection every explore read uses. Before the repair this failed
// with "no such column: total_tracks" on every install that already had
// a catalog, while a fresh one was perfectly healthy.
func TestRetiresIndexMissingAColumn(t *testing.T) {
ctx := context.Background()
dir := t.TempDir()
db := openRaw(t, dir)
// The pre-014 shape: the columns the projection needs, minus the
// one the plan added.
if _, err := db.ExecContext(ctx, `
CREATE TABLE explore_index (
id INTEGER PRIMARY KEY AUTOINCREMENT,
entity_type INTEGER NOT NULL,
mbid BLOB NOT NULL,
title TEXT NOT NULL,
artist_name TEXT NOT NULL,
artist_mbid BLOB NOT NULL
);
INSERT INTO explore_index (entity_type, mbid, title, artist_name, artist_mbid)
VALUES (1, x'00112233445566778899aabbccddeeff', 'x', 'y', x'');
`); err != nil {
t.Fatalf("seed: %v", err)
}
if err := retireStaleTables(ctx, db, testLogger()); err != nil {
t.Fatalf("retire: %v", err)
}
if err := applySchema(ctx, db); err != nil {
t.Fatalf("applySchema: %v", err)
}
// The column the projection needs is there now.
var n int
if err := db.QueryRowContext(ctx,
`SELECT COUNT(*) FROM pragma_table_info('explore_index')
WHERE name = 'total_tracks'`,
).Scan(&n); err != nil {
t.Fatalf("inspect: %v", err)
}
if n != 1 {
t.Fatalf("explore_index still has no total_tracks column")
}
// And the catalog really was retired rather than patched, so the
// artifact is fetched again instead of half a catalog being served.
if err := db.QueryRowContext(ctx,
"SELECT COUNT(*) FROM explore_index",
).Scan(&n); err != nil {
t.Fatalf("count: %v", err)
}
if n != 0 {
t.Fatalf("stale rows survived the retire: %d", n)
}
}
// TestRetiresIndexWithTextMBIDs is the half an ALTER could not have
// repaired: plan 013 changed mbid from TEXT to BLOB, and SQLite does not
// coerce between them, so a query against 16 raw bytes returns no rows
// rather than an error.
func TestRetiresIndexWithTextMBIDs(t *testing.T) {
ctx := context.Background()
dir := t.TempDir()
db := openRaw(t, dir)
if _, err := db.ExecContext(ctx, `
CREATE TABLE explore_index (
id INTEGER PRIMARY KEY AUTOINCREMENT,
entity_type TEXT NOT NULL,
mbid TEXT NOT NULL,
title TEXT NOT NULL,
artist_name TEXT NOT NULL,
artist_mbid TEXT NOT NULL,
total_tracks INTEGER NOT NULL DEFAULT 0
);
`); err != nil {
t.Fatalf("seed: %v", err)
}
if err := retireStaleTables(ctx, db, testLogger()); err != nil {
t.Fatalf("retire: %v", err)
}
if err := applySchema(ctx, db); err != nil {
t.Fatalf("applySchema: %v", err)
}
var typ string
if err := db.QueryRowContext(ctx,
`SELECT type FROM pragma_table_info('explore_index') WHERE name = 'mbid'`,
).Scan(&typ); err != nil {
t.Fatalf("inspect: %v", err)
}
if typ != "BLOB" {
t.Fatalf("mbid is still %s, want BLOB", typ)
}
}
// TestRetiringTheIndexTakesItsMetaWithIt guards the thing that makes the
// repair actually repair: the marker saying the import finished is what
// stops the artifact being fetched again, so a catalog dropped without
// it would stay empty forever.
func TestRetiringTheIndexTakesItsMetaWithIt(t *testing.T) {
ctx := context.Background()
dir := t.TempDir()
db := openRaw(t, dir)
if _, err := db.ExecContext(ctx, `
CREATE TABLE explore_index (
id INTEGER PRIMARY KEY AUTOINCREMENT,
entity_type INTEGER NOT NULL,
mbid BLOB NOT NULL
);
CREATE TABLE explore_index_meta (key TEXT PRIMARY KEY, value TEXT NOT NULL);
INSERT INTO explore_index_meta VALUES ('dump_import_done', '1');
`); err != nil {
t.Fatalf("seed: %v", err)
}
if err := retireStaleTables(ctx, db, testLogger()); err != nil {
t.Fatalf("retire: %v", err)
}
if err := applySchema(ctx, db); err != nil {
t.Fatalf("applySchema: %v", err)
}
var n int
if err := db.QueryRowContext(ctx,
"SELECT COUNT(*) FROM explore_index_meta WHERE key = 'dump_import_done'",
).Scan(&n); err != nil {
t.Fatalf("meta: %v", err)
}
if n != 0 {
t.Fatalf("the import-done marker survived a retired catalog")
}
}
// TestHealthyDatabaseIsUntouched is the other half, and the one that
// would make this dangerous if it failed: a current schema must survive
// a launch with its catalog intact. A repair that retires a healthy
// catalog costs every user an artifact download on every start.
func TestHealthyDatabaseIsUntouched(t *testing.T) {
ctx := context.Background()
dir := t.TempDir()
db := openRaw(t, dir)
if err := applySchema(ctx, db); err != nil {
t.Fatalf("applySchema: %v", err)
}
if _, err := db.ExecContext(ctx, `
INSERT INTO explore_index (entity_type, mbid, title, artist_name, artist_mbid)
VALUES (1, x'00112233445566778899aabbccddeeff', 'x', 'y', x'')
`); err != nil {
t.Fatalf("seed: %v", err)
}
if err := retireStaleTables(ctx, db, testLogger()); err != nil {
t.Fatalf("retire: %v", err)
}
var n int
if err := db.QueryRowContext(ctx,
"SELECT COUNT(*) FROM explore_index",
).Scan(&n); err != nil {
t.Fatalf("count: %v", err)
}
if n != 1 {
t.Fatalf("a healthy catalog was retired: %d rows left", n)
}
}
// TestAuthoredTablesAreNeverRetired states the boundary in a test rather
// than only in a comment: this mechanism deletes data, and the only
// thing standing between it and a user's playlists is the Kind filter.
func TestAuthoredTablesAreNeverRetired(t *testing.T) {
ctx := context.Background()
dir := t.TempDir()
db := openRaw(t, dir)
// A playlists table missing most of its current columns.
if _, err := db.ExecContext(ctx, `
CREATE TABLE playlists (id INTEGER PRIMARY KEY, name TEXT NOT NULL);
INSERT INTO playlists (name) VALUES ('irreplaceable');
`); err != nil {
t.Fatalf("seed: %v", err)
}
if err := retireStaleTables(ctx, db, testLogger()); err != nil {
t.Fatalf("retire: %v", err)
}
var n int
if err := db.QueryRowContext(ctx,
"SELECT COUNT(*) FROM playlists",
).Scan(&n); err != nil {
t.Fatalf("count: %v", err)
}
if n != 1 {
t.Fatalf("an authored table was retired; rows left: %d", n)
}
}
// TestRetiresTablesTheSchemaNoLongerDescribes covers what plan 013 left
// behind on every database that predates it: seven tables the schema
// stopped describing, plus the schema_migrations table that squashing
// the chain retired. They are not stale in shape — they are simply not
// ours any more.
func TestRetiresTablesTheSchemaNoLongerDescribes(t *testing.T) {
ctx := context.Background()
db := openRaw(t, t.TempDir())
if _, err := db.ExecContext(ctx, `
CREATE TABLE recordings (id INTEGER PRIMARY KEY, name TEXT);
CREATE TABLE artist_credit (id INTEGER PRIMARY KEY, text TEXT);
CREATE TABLE schema_migrations (version INTEGER PRIMARY KEY);
`); err != nil {
t.Fatalf("seed: %v", err)
}
if err := retireStaleTables(ctx, db, testLogger()); err != nil {
t.Fatalf("retire: %v", err)
}
for _, table := range []string{"recordings", "artist_credit", "schema_migrations"} {
var n int
if err := db.QueryRowContext(ctx,
"SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name = ?",
table,
).Scan(&n); err != nil {
t.Fatalf("inspect %s: %v", table, err)
}
if n != 0 {
t.Errorf("%s survived; the schema no longer describes it", table)
}
}
}
// TestFTSShadowTablesAreNotObsolete is the sweep's sharp edge: an FTS5
// virtual table is backed by four shadow tables that appear in
// sqlite_master under their own names and are in no schema file.
// Dropping one destroys the index it belongs to.
func TestFTSShadowTablesAreNotObsolete(t *testing.T) {
ctx := context.Background()
db := openRaw(t, t.TempDir())
if err := applySchema(ctx, db); err != nil {
t.Fatalf("applySchema: %v", err)
}
obsolete, err := obsoleteTables(ctx, db)
if err != nil {
t.Fatalf("obsoleteTables: %v", err)
}
if len(obsolete) != 0 {
t.Fatalf("a freshly created schema reported obsolete tables: %v", obsolete)
}
}
// TestRetiringOwnedTablesDoesNotDangle pins the one behaviour that is
// silently wrong rather than loudly broken.
//
// Retiring audio_files leaves playlist entries behind. With
// foreign_keys ON — which applyPRAGMAs has done before NewDB gets here —
// SET NULL fires and they point at nothing. With it OFF they keep ids
// that the rescan reissues from 1, so every playlist quietly fills with
// different songs. Nothing about the schema makes that ordering
// obvious, so it is asserted rather than assumed.
func TestRetiringOwnedTablesDoesNotDangle(t *testing.T) {
ctx := context.Background()
db := openRaw(t, t.TempDir())
if _, err := db.ExecContext(ctx, "PRAGMA foreign_keys = ON"); err != nil {
t.Fatalf("pragma: %v", err)
}
if err := applySchema(ctx, db); err != nil {
t.Fatalf("applySchema: %v", err)
}
// Break audio_files' shape so it is retired, keeping a playlist
// entry that references it.
if _, err := db.ExecContext(ctx, `
INSERT INTO playlists (id, name) VALUES (1, 'keepme');
INSERT INTO libraries (id, name, path) VALUES (0, 'test', '/music');
INSERT INTO audio_files (id, file_path, file_type_id, length_milliseconds)
VALUES (7, '/music/a.flac', 1, 1000);
INSERT INTO playlist_tracks (playlist_id, audio_file_id, position)
VALUES (1, 7, 0);
DROP VIEW IF EXISTS track_metadata;
ALTER TABLE audio_files DROP COLUMN artist_credit;
`); err != nil {
t.Fatalf("seed: %v", err)
}
if err := retireStaleTables(ctx, db, testLogger()); err != nil {
t.Fatalf("retire: %v", err)
}
if err := applySchema(ctx, db); err != nil {
t.Fatalf("applySchema: %v", err)
}
var dangling int
if err := db.QueryRowContext(ctx,
"SELECT COUNT(*) FROM playlist_tracks WHERE audio_file_id IS NOT NULL",
).Scan(&dangling); err != nil {
t.Fatalf("count: %v", err)
}
if dangling != 0 {
t.Fatalf(
"%d playlist entries still point at retired audio_files ids; "+
"a rescan will reissue those ids to different tracks",
dangling,
)
}
// The playlist itself is authored and must be untouched.
var playlists int
if err := db.QueryRowContext(ctx,
"SELECT COUNT(*) FROM playlists",
).Scan(&playlists); err != nil {
t.Fatalf("playlists: %v", err)
}
if playlists != 1 {
t.Fatalf("authored playlist lost: %d", playlists)
}
}
// TestRetiringInterlinkedLegacyTables is the bug the unit tests missed
// and a real database found.
//
// The tables plan 013 retired reference each other -- pre-013
// audio_files has a foreign key into recordings -- so with foreign keys
// ON, dropping them one at a time fails with "FOREIGN KEY constraint
// failed" on whichever goes first, and map iteration order decides
// which that is. Every other test in this file ran with foreign keys
// off and passed happily; the app enables them in applyPRAGMAs before
// the repair runs, so only the real launch path showed it.
func TestRetiringInterlinkedLegacyTables(t *testing.T) {
ctx := context.Background()
db := openRaw(t, t.TempDir())
if _, err := db.ExecContext(ctx, "PRAGMA foreign_keys = ON"); err != nil {
t.Fatalf("pragma: %v", err)
}
// The pre-013 shape, with the reference that makes ordering matter.
// release_group_recordings sorts *after* recordings and references
// it, so the deterministic order retires the parent while the child
// still holds rows pointing at it -- which is the case that fails
// without the deferral, rather than one that fails on some runs.
if _, err := db.ExecContext(ctx, `
CREATE TABLE recordings (id INTEGER PRIMARY KEY, name TEXT);
CREATE TABLE artist_credit (id INTEGER PRIMARY KEY, text TEXT);
CREATE TABLE release_group_recordings (
id INTEGER PRIMARY KEY,
recording_id INTEGER NOT NULL,
FOREIGN KEY(recording_id) REFERENCES recordings(id)
);
CREATE TABLE audio_files (
id INTEGER PRIMARY KEY,
file_path TEXT NOT NULL UNIQUE,
recording_id INTEGER,
FOREIGN KEY(recording_id) REFERENCES recordings(id)
);
INSERT INTO recordings (id, name) VALUES (1, 'x');
INSERT INTO release_group_recordings (id, recording_id) VALUES (1, 1);
INSERT INTO audio_files (id, file_path, recording_id)
VALUES (1, '/music/a.flac', 1);
`); err != nil {
t.Fatalf("seed: %v", err)
}
if err := retireStaleTables(ctx, db, testLogger()); err != nil {
t.Fatalf("retire: %v", err)
}
if err := applySchema(ctx, db); err != nil {
t.Fatalf("applySchema: %v", err)
}
for _, table := range []string{"recordings", "artist_credit"} {
var n int
if err := db.QueryRowContext(ctx,
"SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name = ?",
table,
).Scan(&n); err != nil {
t.Fatalf("inspect %s: %v", table, err)
}
if n != 0 {
t.Errorf("%s survived the retire", table)
}
}
// And the rebuilt audio_files is the current shape, which is the
// whole reason the old one had to go.
var n int
if err := db.QueryRowContext(ctx,
`SELECT COUNT(*) FROM pragma_table_info('audio_files')
WHERE name = 'artist_credit'`,
).Scan(&n); err != nil {
t.Fatalf("inspect audio_files: %v", err)
}
if n != 1 {
t.Fatal("audio_files was not rebuilt in the current shape")
}
}
// TestParseCreateTablesReadsTheRealSchema keeps the parser honest
// against the files it actually runs on: a parser that silently found
// no columns would report every table healthy and repair nothing.
func TestParseCreateTablesReadsTheRealSchema(t *testing.T) {
declared, err := declaredTables()
if err != nil {
t.Fatalf("declaredTables: %v", err)
}
cols, ok := declared["explore_index"]
if !ok {
t.Fatal("explore_index was not parsed out of the schema files")
}
want := map[string]string{
"mbid": "BLOB",
"total_tracks": "INTEGER",
"artist_name": "TEXT",
}
got := make(map[string]string, len(cols))
for _, c := range cols {
got[c.name] = c.typ
}
for name, typ := range want {
if got[name] != typ {
t.Errorf("explore_index.%s parsed as %q, want %q", name, got[name], typ)
}
}
// A table constraint must not be mistaken for a column.
for _, c := range cols {
switch c.name {
case "PRIMARY", "FOREIGN", "UNIQUE", "CHECK", "CONSTRAINT":
t.Errorf("parsed table constraint %q as a column", c.name)
}
}
}
+15
View File
@@ -185,6 +185,21 @@ var tables = []Table{
Note: "Full-text index over the champion entities of the " +
"MusicBrainz dump. Rebuilt only by a full index build.",
},
{
Name: "artist_credit_part", Kind: Cache, Lifetime: Retained,
Note: "The decomposition of a multi-artist credit, from the " +
"MusicBrainz dump: one row per credited artist, with the " +
"name as credited and the join phrase that follows it. " +
"Arrives with the downloaded artifact, so rebuilding it " +
"costs a dump stream and it is never swept.",
},
{
Name: "artist_credit_ref", Kind: Cache, Lifetime: Retained,
Note: "Which credit a catalog recording or release group is " +
"credited to. Present only for multi-artist credits; " +
"absence means one artist, which explore_index already " +
"describes. Ships and dies with artist_credit_part.",
},
{
Name: "explore_index", Kind: Cache, Lifetime: Retained,
Note: "The offline MusicBrainz search index. Rebuilding costs a " +
+15 -16
View File
@@ -199,24 +199,23 @@ func TestManagerEndToEndAutoPick(t *testing.T) {
t.Errorf("expected imported file at %s: %v", want, err)
}
// Staging was released only after a successful import.
entries, err := os.ReadDir(f.staging.Root())
if err != nil {
t.Fatalf("read staging root: %v", err)
}
// Staging release and the rescan happen *after* the state is
// recorded (manager.go sets StateComplete, then releases, then
// scans), so waiting on the state is not waiting on these. Under
// load the worker is descheduled in between and asserting straight
// away reads the world one step too early -- which is exactly how
// this test failed on a busy machine while passing alone.
waitFor(t, func() bool {
entries, err := os.ReadDir(f.staging.Root())
if err != nil || len(entries) != 0 {
return false
}
if len(entries) != 0 {
t.Errorf("staging not released: %d dirs remain", len(entries))
}
f.lib.mu.Lock()
defer f.lib.mu.Unlock()
// The library was told to rescan.
f.lib.mu.Lock()
scanned := len(f.lib.scanned)
f.lib.mu.Unlock()
if scanned != 1 {
t.Errorf("library scans = %d, want 1", scanned)
}
return len(f.lib.scanned) == 1
}, "staging was never released, or the library was never rescanned")
}
// An ambiguous result set must park for the user rather than guess.
+75
View File
@@ -283,6 +283,9 @@ func (si *SearchIndex) importCoreArtifact(ctx context.Context, path string) erro
}
merged, mergeErr := si.mergeArtifactRows(ctx, info.rows)
if mergeErr == nil {
si.mergeArtifactCredits(ctx)
}
if ftsSuspended {
start := time.Now()
@@ -473,3 +476,75 @@ func (si *SearchIndex) removeArtifactFile(path string) {
si.logger.Warn("core artifact: cleanup failed", "path", path, "error", err)
}
}
// artifactHasCredits reports whether the attached artifact carries the
// multi-artist credit tables.
//
// The same shape, and the same handle, as artifactHasTotals above: an
// artifact published before credits existed is still a perfectly good
// catalog, and there is one already out there. Selecting from a table
// that is not in it would fail an import that should have succeeded, so
// it is asked rather than assumed -- on the *writer*, because `core` is
// attached to that one connection and the read pool cannot see it.
func (si *SearchIndex) artifactHasCredits() bool {
var n int
err := si.db.QueryRowWriter(
`SELECT COUNT(*) FROM core.sqlite_master
WHERE type = 'table' AND name IN ('artist_credit_part', 'artist_credit_ref')`,
).Scan(&n)
return err == nil && n == 2
}
// mergeArtifactCredits copies the credit decomposition out of the
// attached artifact.
//
// Credits are replaced wholesale rather than merged: they are derived
// entirely from one dump build, they are keyed by ids that are only
// meaningful within the artifact that carried them, and a half-updated
// credit renders as the wrong artists rather than as missing ones.
//
// A failure here is logged and not returned. The catalog has already
// merged at this point, and a catalog without credits is the catalog
// this app had before them -- every credit falls back to its single
// artist, which is the same fallback an untagged file already gets.
func (si *SearchIndex) mergeArtifactCredits(ctx context.Context) {
if !si.artifactHasCredits() {
si.logger.Info("core artifact: no credit tables, keeping single-artist credits")
return
}
start := time.Now()
for _, stmt := range []string{
"DELETE FROM artist_credit_part",
"DELETE FROM artist_credit_ref",
`INSERT OR REPLACE INTO artist_credit_part
(credit_id, position, artist_mbid, credited_name, join_phrase)
SELECT credit_id, position, artist_mbid, credited_name, join_phrase
FROM core.artist_credit_part`,
`INSERT OR REPLACE INTO artist_credit_ref (mbid, credit_id)
SELECT mbid, credit_id FROM core.artist_credit_ref`,
} {
if err := ctx.Err(); err != nil {
return
}
if _, err := si.db.ExecContext(stmt); err != nil {
si.logger.Warn("core artifact: credit merge failed", "error", err)
return
}
}
var refs int
_ = si.db.QueryRowWriter("SELECT COUNT(*) FROM artist_credit_ref").Scan(&refs)
si.logger.Info("core artifact: credits merged",
"entities", refs,
"elapsed", time.Since(start).Round(time.Millisecond),
)
}
+151
View File
@@ -3,6 +3,7 @@ package explore
import (
"context"
"database/sql"
"encoding/hex"
"os"
"path/filepath"
"strings"
@@ -597,3 +598,153 @@ func TestImportCoreArtifactReadsTotalsWhenPresent(t *testing.T) {
t.Errorf("TotalTracks = %d, want 0 (the catalog does not say)", old.TotalTracks)
}
}
// addArtifactCredits gives an artifact file the credit tables the
// exporter now writes, so the import path can be exercised against one
// that has them.
func addArtifactCredits(t *testing.T, path string) {
t.Helper()
db, err := sql.Open("sqlite", "file:"+path)
if err != nil {
t.Fatalf("open artifact: %v", err)
}
defer func() { _ = db.Close() }()
for _, stmt := range []string{
`CREATE TABLE artist_credit_part (
credit_id INTEGER NOT NULL,
position INTEGER NOT NULL,
artist_mbid BLOB NOT NULL,
credited_name TEXT NOT NULL,
join_phrase TEXT NOT NULL DEFAULT '',
PRIMARY KEY (credit_id, position)
) WITHOUT ROWID`,
`CREATE TABLE artist_credit_ref (
mbid BLOB NOT NULL PRIMARY KEY,
credit_id INTEGER NOT NULL
) WITHOUT ROWID`,
} {
if _, err := db.Exec(stmt); err != nil {
t.Fatalf("create credit tables: %v", err)
}
}
// The packed form the catalog stores. uuid16/parseUUID live behind
// the indexbuild tag, so this file decodes for itself.
pack := func(mbid string) []byte {
raw, err := hex.DecodeString(strings.ReplaceAll(mbid, "-", ""))
if err != nil || len(raw) != 16 {
t.Fatalf("fixture MBID %q is not a UUID: %v", mbid, err)
}
return raw
}
a, b, rec := pack(artA), pack(artB), pack(recA)
for _, part := range [][]any{
{7, 0, a, "Artist A", " feat. "},
{7, 1, b, "Artist B", ""},
} {
if _, err := db.Exec(`INSERT INTO artist_credit_part
(credit_id, position, artist_mbid, credited_name, join_phrase)
VALUES (?, ?, ?, ?, ?)`, part...); err != nil {
t.Fatalf("insert part: %v", err)
}
}
if _, err := db.Exec(
"INSERT INTO artist_credit_ref (mbid, credit_id) VALUES (?, ?)", rec, 7,
); err != nil {
t.Fatalf("insert ref: %v", err)
}
}
// TestImportCoreArtifactMergesCredits is the positive half of the
// compatibility pair: an artifact that carries credits delivers them,
// rendering back to the credit string they decompose.
func TestImportCoreArtifactMergesCredits(t *testing.T) {
db := database.NewTestDB(t)
si := NewSearchIndex(db, nil, nil, testLogger())
path := writeTestArtifact(t, validMeta(), []artifactRow{
{"recording", recA, "Song A", "Artist A feat. Artist B", artA, 2000},
})
addArtifactCredits(t, path)
if err := si.importCoreArtifact(context.Background(), path); err != nil {
t.Fatalf("importCoreArtifact: %v", err)
}
rows, err := db.QueryContext(
`SELECT p.credited_name, p.join_phrase
FROM artist_credit_ref r
JOIN artist_credit_part p ON p.credit_id = r.credit_id
ORDER BY p.position`,
)
if err != nil {
t.Fatalf("query credits: %v", err)
}
defer func() { _ = rows.Close() }()
var rendered strings.Builder
for rows.Next() {
var name, join string
if err := rows.Scan(&name, &join); err != nil {
t.Fatalf("scan: %v", err)
}
rendered.WriteString(name)
rendered.WriteString(join)
}
if got := rendered.String(); got != "Artist A feat. Artist B" {
t.Errorf("rendered credit = %q, want %q", got, "Artist A feat. Artist B")
}
}
// TestImportCoreArtifactWithoutCredits is the regression that matters
// most here: an artifact published before credits existed cannot be
// re-cut retroactively, so it must import as a catalog that declines to
// answer rather than failing outright. writeTestArtifact deliberately
// builds one without the tables.
func TestImportCoreArtifactWithoutCredits(t *testing.T) {
db := database.NewTestDB(t)
si := NewSearchIndex(db, nil, nil, testLogger())
path := writeTestArtifact(t, validMeta(), []artifactRow{
{"recording", recA, "Song A", "Artist A", artA, 2000},
})
if err := si.importCoreArtifact(context.Background(), path); err != nil {
t.Fatalf("an artifact without credit tables must still import: %v", err)
}
var rows int
if err := db.QueryRowWriter(
"SELECT COUNT(*) FROM explore_index",
).Scan(&rows); err != nil {
t.Fatalf("count: %v", err)
}
if rows != 1 {
t.Errorf("catalog rows = %d, want 1", rows)
}
var refs int
if err := db.QueryRowWriter(
"SELECT COUNT(*) FROM artist_credit_ref",
).Scan(&refs); err != nil {
t.Fatalf("count refs: %v", err)
}
if refs != 0 {
t.Errorf("credit refs = %d, want 0", refs)
}
}
+132
View File
@@ -0,0 +1,132 @@
package explore
import (
"fmt"
"strings"
)
// Reading multi-artist credits back out of the catalog.
//
// The tables are filled centrally (backend/explore/dumpcredits.go, and
// the artifact import) and hold only credits naming more than one
// artist: an entity with no rows here is credited to one artist, which
// explore_index's own artist_name and artist_mbid already describe.
// Absence is the common case and means "nothing to decompose", never
// "unknown".
//
// The lookup is keyed on the *recording* MBID, which both sides of the
// app already have -- a catalog row carries it and so does a local
// file (library.Track.RecordingMBID) -- so one query serves the Explore
// pages and the library's own lists without either needing to know
// where the other gets its rows.
// CreditPart is one credited artist within a credit, in credit order.
//
// CreditedName is the name *as credited*, which is not the artist's own
// name: MusicBrainz credits "Snoop Dogg" on a track by the artist
// called "Snoop Doggy Dogg". Display uses it; navigation uses
// ArtistMBID. JoinPhrase is the literal connector that follows this
// part, so a credit renders by concatenation and never by searching a
// name inside a credit string.
type CreditPart struct {
Position int `json:"position"`
ArtistMBID string `json:"artistMbid"`
CreditedName string `json:"creditedName"`
JoinPhrase string `json:"joinPhrase"`
}
// creditLookupBatch bounds how many MBIDs go into one IN clause. A
// tracklist is the caller here, so the realistic ceiling is a few
// hundred; the bound exists so a 50,000-row selection cannot build a
// statement SQLite refuses to parse.
const creditLookupBatch = 500
// GetCredits returns the decomposition of every multi-artist credit
// among the given entity MBIDs, keyed by MBID.
//
// MBIDs with a single-artist credit are simply absent from the result,
// which is what the caller wants: it renders its existing single link
// for those, and that is the same answer it would have rendered anyway.
func (si *SearchIndex) GetCredits(mbids []string) (map[string][]CreditPart, error) {
out := make(map[string][]CreditPart)
for start := 0; start < len(mbids); start += creditLookupBatch {
end := min(start+creditLookupBatch, len(mbids))
if err := si.appendCredits(mbids[start:end], out); err != nil {
return nil, err
}
}
return out, nil
}
// appendCredits runs one batch into the accumulating result.
func (si *SearchIndex) appendCredits(
mbids []string, out map[string][]CreditPart,
) error {
args := make([]any, 0, len(mbids))
holders := make([]string, 0, len(mbids))
for _, mbid := range mbids {
if mbid == "" {
continue
}
args = append(args, dbMBID(mbid))
holders = append(holders, "?")
}
if len(args) == 0 {
return nil
}
// Ordered by position because that ordering *is* the credit's
// meaning; the caller concatenates in the order it receives.
rows, err := si.db.QueryContext(
`SELECT r.mbid, p.position, p.artist_mbid, p.credited_name, p.join_phrase
FROM artist_credit_ref r
JOIN artist_credit_part p ON p.credit_id = r.credit_id
WHERE r.mbid IN (`+strings.Join(holders, ",")+`)
ORDER BY r.mbid, p.position`,
args...,
)
if err != nil {
return fmt.Errorf("read artist credits: %w", err)
}
defer func() { _ = rows.Close() }()
for rows.Next() {
var (
entity dbMBID
artist dbMBID
part CreditPart
)
if err := rows.Scan(
&entity, &part.Position, &artist, &part.CreditedName, &part.JoinPhrase,
); err != nil {
return fmt.Errorf("scan artist credit: %w", err)
}
part.ArtistMBID = string(artist)
out[string(entity)] = append(out[string(entity)], part)
}
if err := rows.Err(); err != nil {
return fmt.Errorf("read artist credits: %w", err)
}
return nil
}
// GetCredits is the bound form: the frontend asks for a tracklist's
// worth of MBIDs at once rather than one per row.
//
// Batched for the reason every other per-row backend question here is:
// asking on hover or on render turns a list into N IPC round trips, and
// this one is asked about every row of every list in the app.
func (e *Service) GetCredits(mbids []string) (map[string][]CreditPart, error) {
return e.index.GetCredits(mbids)
}
+128
View File
@@ -0,0 +1,128 @@
package explore
import (
"encoding/hex"
"fmt"
"strings"
"testing"
"yellowjacket/backend/database"
)
// seedCredit writes one multi-artist credit and points an entity at it,
// the way the dump import and the artifact import both do.
func seedCredit(t *testing.T, db *database.DB, entity string, id int, parts []CreditPart) {
t.Helper()
pack := func(mbid string) []byte {
raw, err := hex.DecodeString(strings.ReplaceAll(mbid, "-", ""))
if err != nil || len(raw) != 16 {
t.Fatalf("bad fixture mbid %q: %v", mbid, err)
}
return raw
}
if _, err := db.ExecContext(
"INSERT INTO artist_credit_ref (mbid, credit_id) VALUES (?, ?)",
pack(entity), id,
); err != nil {
t.Fatalf("seed ref: %v", err)
}
for _, p := range parts {
if _, err := db.ExecContext(
`INSERT INTO artist_credit_part
(credit_id, position, artist_mbid, credited_name, join_phrase)
VALUES (?, ?, ?, ?, ?)`,
id, p.Position, pack(p.ArtistMBID), p.CreditedName, p.JoinPhrase,
); err != nil {
t.Fatalf("seed part: %v", err)
}
}
}
// TestGetCreditsDecomposes: the parts come back in position order and
// concatenate to the credit they describe.
func TestGetCreditsDecomposes(t *testing.T) {
db := database.NewTestDB(t)
si := NewSearchIndex(db, nil, nil, testLogger())
rec := testMBID("rec-1")
a, b := testMBID("artist-a"), testMBID("artist-b")
seedCredit(t, db, rec, 7, []CreditPart{
{Position: 0, ArtistMBID: a, CreditedName: "2Pac", JoinPhrase: " feat. "},
{Position: 1, ArtistMBID: b, CreditedName: "Snoop Dogg"},
})
got, err := si.GetCredits([]string{rec})
if err != nil {
t.Fatalf("GetCredits: %v", err)
}
parts := got[rec]
if len(parts) != 2 {
t.Fatalf("parts = %d, want 2", len(parts))
}
var rendered strings.Builder
for _, p := range parts {
rendered.WriteString(p.CreditedName)
rendered.WriteString(p.JoinPhrase)
}
if rendered.String() != "2Pac feat. Snoop Dogg" {
t.Errorf("rendered = %q, want %q", rendered.String(), "2Pac feat. Snoop Dogg")
}
// Dashed on the way out: a blob reaching the frontend is sixteen
// bytes of mojibake, and nothing above mbid.go speaks that.
if parts[0].ArtistMBID != a {
t.Errorf("artist mbid = %q, want %q", parts[0].ArtistMBID, a)
}
}
// TestGetCreditsOmitsSingleArtist: absence is the common case and means
// "nothing to decompose", so the caller renders its existing one link.
func TestGetCreditsOmitsSingleArtist(t *testing.T) {
db := database.NewTestDB(t)
si := NewSearchIndex(db, nil, nil, testLogger())
got, err := si.GetCredits([]string{testMBID("untagged"), ""})
if err != nil {
t.Fatalf("GetCredits: %v", err)
}
if len(got) != 0 {
t.Errorf("got %d credits, want none", len(got))
}
}
// TestGetCreditsBatches: the lookup is asked about whole tracklists, so
// it must not build one statement per row or one SQLite refuses to
// parse.
func TestGetCreditsBatches(t *testing.T) {
db := database.NewTestDB(t)
si := NewSearchIndex(db, nil, nil, testLogger())
mbids := make([]string, 0, creditLookupBatch*2+7)
for i := range creditLookupBatch*2 + 7 {
mbids = append(mbids, testMBID(fmt.Sprintf("batch-%d", i)))
}
// One real credit somewhere past the first batch boundary.
seedCredit(t, db, mbids[creditLookupBatch+3], 9, []CreditPart{
{Position: 0, ArtistMBID: testMBID("a"), CreditedName: "A", JoinPhrase: " & "},
{Position: 1, ArtistMBID: testMBID("b"), CreditedName: "B"},
})
got, err := si.GetCredits(mbids)
if err != nil {
t.Fatalf("GetCredits: %v", err)
}
if len(got[mbids[creditLookupBatch+3]]) != 2 {
t.Errorf("a credit past the first batch boundary was not returned")
}
}
+637
View File
@@ -0,0 +1,637 @@
//go:build indexbuild
package explore
import (
"archive/tar"
"bufio"
"compress/bzip2"
"context"
"errors"
"fmt"
"io"
"path"
"regexp"
"strconv"
"strings"
)
// Multi-artist credits, from the core MusicBrainz dump.
//
// A credit is ordered parts and the credit *string* is derived from
// them; MusicBrainz's own artist_credit.name is a cached render. What
// this pass extracts is the decomposition: for each catalog recording
// and release group whose credit names more than one artist, the
// credited artists in order, each with the name *as credited* and the
// join phrase that follows it. See artist_credit_part.sql for why that
// is stored rather than derived, and why nothing may reconstruct a
// credit by searching a name inside a credit string.
//
// It is a separate dump from everything else here, and it has to be.
// The canonical dump this importer already streams gives artist_mbids
// (an ordered list) and artist_credit_name (the *rendered* string) --
// no join phrases, and no per-artist as-credited names. Splitting the
// rendered string using canonical artist names fails on exactly the
// credits that matter: measured on a real library, 21% of multi-artist
// credits name an artist differently from the artist's own name
// ("Snoop Dogg" credited on a track by "Snoop Doggy Dogg"), so the
// substring is simply not there. The JSON dumps were checked too and
// cover 153,691 recordings of ~35M, with zero overlap against a real
// library. This dump is the only source.
//
// Cost, measured on the 20260815 export: 7.1 GB compressed, decompressed
// by pure-Go compress/bzip2 at ~26 MB/s uncompressed (~13.7 min for the
// whole file, single-threaded). cmd/indexbuild is built CGO_ENABLED=0,
// so the stdlib decompressor is what there is -- and it is fine, because
// the 2 MB/s origin throttle dominates, as it does for every other dump
// here.
const (
// defaultMBDumpBaseURL is the core MusicBrainz export. Only
// mbdump.tar.bz2 is fetched; the other tarballs there hold data this
// app has no use for.
defaultMBDumpBaseURL = "https://data.metabrainz.org/pub/musicbrainz/data/fullexport/"
)
var (
mbdumpDirRe = regexp.MustCompile(`^\d{8}-\d+$`)
mbdumpFileRe = regexp.MustCompile(`^mbdump\.tar\.bz2$`)
// ErrDumpShape is returned when a dump member does not have the
// columns this code was written against. It is deliberately fatal:
// reading the wrong column silently produces a catalog whose credits
// are subtly wrong, which is far worse than a failed build.
ErrDumpShape = errors.New("musicbrainz dump member has an unexpected shape")
)
// Column positions in the Postgres COPY output, verified against the
// 20260815 export. There is no header row to read them from, so they
// are asserted instead -- see checkShape.
const (
artistColID = 0
artistColGID = 1
artistColMin = 2
creditColID = 0
creditColArtistCount = 2
creditColMin = 3
partColCredit = 0
partColPosition = 1
partColArtist = 2
partColName = 3
partColJoin = 4
partColMin = 5
// recording and release_group share a layout in the columns this
// pass reads: id, gid, name, artist_credit, ...
entityColGID = 1
entityColCredit = 3
entityColMin = 4
)
// creditPart is one credited artist within a credit.
type creditPart struct {
position int
artistID int32
name string
join string
}
// creditScan is what one pass over the dump collects.
type creditScan struct {
// artistGIDs maps an artist row id to its MBID. artist_credit_name
// references artists by row id, and the tar orders `artist` before
// it, so this is complete by the time it is read.
artistGIDs map[int32]uuid16
// multiCredits are the credit ids naming more than one artist, from
// artist_credit.artist_count. Taking the count from the dump rather
// than counting parts means a credit can be rejected before its
// parts are stored.
multiCredits map[int32]struct{}
// parts are the decompositions of multiCredits, keyed by credit id.
parts map[int32][]creditPart
// refs maps a kept catalog entity to its credit. Only entities in
// explore_index and only multi-artist credits: everything else is
// already described by explore_index's own artist_name/artist_mbid.
refs map[uuid16]int32
// used are the credits some ref actually points at, which is a small
// fraction of multiCredits -- the catalog keeps ~1.8M entities of
// MusicBrainz's tens of millions.
used map[int32]struct{}
skippedUnknownArtist int
}
// creditsImportDoneKey marks in explore_index_meta that the credit pass
// has run against the current catalog.
//
// It is its own marker rather than part of the import's stage state for
// a resume reason: the credit pass runs *after* the catalog is
// assembled, and a failure in it must not send the next run back
// through the ~205 GB it just finished. Marking separately means a
// retry retries only this.
const creditsImportDoneKey = "credits_import_done"
// ensureArtistCredits runs the credit pass unless it has already run
// against this catalog, reporting whether it newly populated them.
//
// Called from both of run's paths -- the full import and the resume
// that finds the rows already assembled -- and from the maintenance
// entry point below, since a catalog built before credits existed is
// otherwise never offered a chance to gain them: the index job picks
// its mode from the index's own state, and a complete import means
// "refresh", which never enters run() at all.
//
// The return value is what tells the job there is something new worth
// publishing. A refresh otherwise reports "changed" only when the
// listens series advanced, so credits would sit in the CI database and
// never reach an artifact.
func (imp *dumpImporter) ensureArtistCredits(ctx context.Context) bool {
if imp.si.hasMeta(creditsImportDoneKey) {
return false
}
url, err := discoverDumpFile(
ctx, imp.httpClient, imp.mbdumpBaseURL, mbdumpDirRe, mbdumpFileRe,
)
if err != nil {
imp.logger.Warn("credit import: could not find the dump", "error", err)
return false
}
if err := imp.importArtistCredits(ctx, url); err != nil {
// A catalog without credits is the catalog this app shipped
// before them: every credit falls back to its single artist.
// That is worth far less than failing an import that otherwise
// succeeded.
imp.logger.Warn("credit import: failed", "error", err)
return false
}
imp.si.setMeta(creditsImportDoneKey, "1")
return true
}
// EnsureArtistCredits tops up the credit tables outside a full import.
//
// It exists because the index job's modes are decided from the index's
// own state: a cache holding a completed import chooses `refresh`,
// which folds in incremental listens and never enters the dump
// importer. Without this, a catalog built before the credit pass
// existed could only gain credits from a `rebuild` -- and a rebuild
// re-downloads ~205 GB to reproduce rows it already has, to add
// something that costs 7 GB on its own.
//
// Reports whether credits were newly populated, so the caller knows
// there is a new artifact worth publishing.
func (e *Service) EnsureArtistCredits(ctx context.Context) bool {
imp, err := newDumpImporter(e.index, e.lb)
if err != nil {
e.index.logger.Warn("credit import: could not start", "error", err)
return false
}
return imp.ensureArtistCredits(ctx)
}
// importArtistCredits streams the core MusicBrainz dump and fills
// artist_credit_part and artist_credit_ref for the entities the catalog
// kept.
//
// It runs after assembleIndex because it asks explore_index which
// entities those are: the popularity filter decides what is worth
// carrying credits for, and asking the table rather than the kept sets
// means this stays correct if that filter changes.
func (imp *dumpImporter) importArtistCredits(ctx context.Context, url string) error {
kept, err := imp.keptEntityMBIDs(ctx)
if err != nil {
return err
}
if len(kept) == 0 {
imp.logger.Warn("credit import: no catalog entities, skipping")
return nil
}
imp.logger.Info("credit import: starting", "url", url, "entities", len(kept))
imp.logJob("Streaming MusicBrainz dump for artist credits")
scan, err := imp.scanCreditDump(ctx, url, kept)
if err != nil {
return err
}
imp.logger.Info("credit import: scanned",
"multiArtistCredits", len(scan.multiCredits),
"entitiesWithMultiArtistCredit", len(scan.refs),
"creditsUsed", len(scan.used),
)
return imp.writeCredits(ctx, scan)
}
// keptEntityMBIDs is every recording and release group in the catalog.
// Artists are excluded: an artist is not credited to a credit.
func (imp *dumpImporter) keptEntityMBIDs(ctx context.Context) (map[uuid16]struct{}, error) {
rows, err := imp.si.db.QueryContextWith(ctx,
`SELECT mbid FROM explore_index
WHERE entity_type IN (2 /* release_group */, 3 /* recording */)`,
)
if err != nil {
return nil, fmt.Errorf("credit import: read catalog entities: %w", err)
}
defer func() { _ = rows.Close() }()
out := make(map[uuid16]struct{})
for rows.Next() {
var raw []byte
if err := rows.Scan(&raw); err != nil {
return nil, fmt.Errorf("credit import: scan mbid: %w", err)
}
if len(raw) != len(uuid16{}) {
continue
}
var id uuid16
copy(id[:], raw)
out[id] = struct{}{}
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("credit import: read catalog entities: %w", err)
}
return out, nil
}
// scanCreditDump makes one sequential pass over mbdump.tar.bz2.
//
// The tar's members are alphabetical, which is what makes a single pass
// possible without buffering the big ones: `artist` and
// `artist_credit_name` both arrive before `recording` and
// `release_group`, so by the time an entity names a credit, that
// credit's parts and their artists' MBIDs are already known and the
// entity can be resolved and dropped. 35M recording rows are never
// held.
//
// The order is not depended on blindly: an entity naming a credit that
// has not been seen is counted and reported rather than silently
// producing an empty catalog, which is what a reordered export would
// otherwise look like.
func (imp *dumpImporter) scanCreditDump(
ctx context.Context, url string, kept map[uuid16]struct{},
) (*creditScan, error) {
stream := imp.openDumpStream(ctx, url, 0)
defer func() { _ = stream.Close() }()
return imp.scanCreditTar(
ctx,
tar.NewReader(bzip2.NewReader(bufio.NewReaderSize(stream, 1<<20))),
kept,
)
}
// scanCreditTar is the parse, separated from the fetch so it can be
// driven by a tar built in a test. compress/bzip2 is decompress-only,
// so a test cannot produce the real container.
func (imp *dumpImporter) scanCreditTar(
ctx context.Context, tr *tar.Reader, kept map[uuid16]struct{},
) (*creditScan, error) {
scan := &creditScan{
artistGIDs: make(map[int32]uuid16),
multiCredits: make(map[int32]struct{}),
parts: make(map[int32][]creditPart),
refs: make(map[uuid16]int32),
used: make(map[int32]struct{}),
}
for {
if err := ctx.Err(); err != nil {
return nil, err
}
hdr, err := tr.Next()
if errors.Is(err, io.EOF) {
break
}
if err != nil {
return nil, fmt.Errorf("credit import: tar: %w", err)
}
if hdr.Typeflag != tar.TypeReg {
continue
}
done, err := imp.scanCreditMember(ctx, hdr.Name, tr, kept, scan)
if err != nil {
return nil, err
}
if done {
// Everything this pass needs has been read; the rest of the
// tarball is other entities' data and decompressing it would
// cost minutes for nothing.
break
}
}
if scan.skippedUnknownArtist > 0 {
imp.logger.Warn("credit import: credits dropped for unknown artists",
"count", scan.skippedUnknownArtist,
)
}
return scan, nil
}
// scanCreditMember dispatches one tar member, reporting whether the
// pass has everything it needs.
func (imp *dumpImporter) scanCreditMember(
ctx context.Context, name string, r io.Reader,
kept map[uuid16]struct{}, scan *creditScan,
) (bool, error) {
switch path.Base(name) {
case "artist":
return false, imp.scanArtists(ctx, r, scan)
case "artist_credit":
return false, imp.scanCredits(ctx, r, scan)
case "artist_credit_name":
return false, imp.scanCreditParts(ctx, r, scan)
case "recording", "release_group":
if err := imp.scanCreditedEntities(ctx, r, kept, scan); err != nil {
return false, err
}
// release_group sorts after recording, so the pass is complete
// once it has been read.
return path.Base(name) == "release_group", nil
default:
return false, nil
}
}
// scanArtists records every artist's MBID by row id.
func (imp *dumpImporter) scanArtists(
ctx context.Context, r io.Reader, scan *creditScan,
) error {
return scanTSV(ctx, r, artistColMin, "artist", func(fields []string) error {
id, ok := parseInt32(fields[artistColID])
if !ok {
return nil
}
var gid uuid16
if !parseUUID(fields[artistColGID], gid[:]) {
return fmt.Errorf("%w: artist.gid is not a UUID: %q",
ErrDumpShape, truncate(fields[artistColGID]))
}
scan.artistGIDs[id] = gid
return nil
})
}
// scanCredits records which credits name more than one artist.
func (imp *dumpImporter) scanCredits(
ctx context.Context, r io.Reader, scan *creditScan,
) error {
return scanTSV(ctx, r, creditColMin, "artist_credit", func(fields []string) error {
id, ok := parseInt32(fields[creditColID])
if !ok {
return nil
}
count, ok := parseInt32(fields[creditColArtistCount])
if !ok {
return fmt.Errorf("%w: artist_credit.artist_count is not a number: %q",
ErrDumpShape, truncate(fields[creditColArtistCount]))
}
if count > 1 {
scan.multiCredits[id] = struct{}{}
}
return nil
})
}
// scanCreditParts records the decomposition of every multi-artist
// credit.
func (imp *dumpImporter) scanCreditParts(
ctx context.Context, r io.Reader, scan *creditScan,
) error {
return scanTSV(ctx, r, partColMin, "artist_credit_name", func(fields []string) error {
credit, ok := parseInt32(fields[partColCredit])
if !ok {
return nil
}
if _, multi := scan.multiCredits[credit]; !multi {
return nil
}
position, ok := parseInt32(fields[partColPosition])
if !ok {
return nil
}
artist, ok := parseInt32(fields[partColArtist])
if !ok {
return nil
}
scan.parts[credit] = append(scan.parts[credit], creditPart{
position: int(position),
artistID: artist,
name: fields[partColName],
join: fields[partColJoin],
})
return nil
})
}
// scanCreditedEntities resolves recordings and release groups against
// the catalog, keeping only those the catalog holds and whose credit
// names more than one artist.
func (imp *dumpImporter) scanCreditedEntities(
ctx context.Context, r io.Reader, kept map[uuid16]struct{}, scan *creditScan,
) error {
return scanTSV(ctx, r, entityColMin, "recording/release_group",
func(fields []string) error {
var gid uuid16
if !parseUUID(fields[entityColGID], gid[:]) {
return fmt.Errorf("%w: entity gid is not a UUID: %q",
ErrDumpShape, truncate(fields[entityColGID]))
}
if _, want := kept[gid]; !want {
return nil
}
credit, ok := parseInt32(fields[entityColCredit])
if !ok {
return fmt.Errorf("%w: entity artist_credit is not a number: %q",
ErrDumpShape, truncate(fields[entityColCredit]))
}
if _, multi := scan.multiCredits[credit]; !multi {
return nil
}
scan.refs[gid] = credit
scan.used[credit] = struct{}{}
return nil
})
}
// scanTSV reads Postgres COPY output a line at a time, unescaping each
// field and handing the row to fn.
//
// The shape is asserted on the first row rather than trusted: this dump
// has no header, so a column that moved would otherwise be read as a
// neighbouring one and produce a catalog that is quietly wrong.
func scanTSV(
ctx context.Context, r io.Reader, minCols int, member string,
fn func(fields []string) error,
) error {
sc := bufio.NewScanner(r)
sc.Buffer(make([]byte, 0, 1<<20), 1<<24)
checked := false
rows := 0
for sc.Scan() {
rows++
if rows%(1<<20) == 0 {
if err := ctx.Err(); err != nil {
return err
}
}
line := sc.Text()
if line == "" {
continue
}
fields := strings.Split(line, "\t")
if len(fields) < minCols {
if !checked {
return fmt.Errorf("%w: %s has %d columns, need at least %d",
ErrDumpShape, member, len(fields), minCols)
}
continue
}
checked = true
for i := range fields {
fields[i] = unescapeCopy(fields[i])
}
if err := fn(fields); err != nil {
return err
}
}
if err := sc.Err(); err != nil {
return fmt.Errorf("credit import: read %s: %w", member, err)
}
return nil
}
// unescapeCopy undoes Postgres COPY's text escaping. A NULL (\N) is
// returned as an empty string: every field this pass reads is either a
// number it will reject or a name whose absence means the same as
// empty.
func unescapeCopy(s string) string {
if s == `\N` {
return ""
}
if !strings.ContainsRune(s, '\\') {
return s
}
var b strings.Builder
b.Grow(len(s))
for i := 0; i < len(s); i++ {
if s[i] != '\\' || i+1 >= len(s) {
b.WriteByte(s[i])
continue
}
i++
switch s[i] {
case 'n':
b.WriteByte('\n')
case 't':
b.WriteByte('\t')
case 'r':
b.WriteByte('\r')
case 'b':
b.WriteByte('\b')
case 'f':
b.WriteByte('\f')
case 'v':
b.WriteByte('\v')
case '\\':
b.WriteByte('\\')
default:
b.WriteByte('\\')
b.WriteByte(s[i])
}
}
return b.String()
}
func parseInt32(s string) (int32, bool) {
n, err := strconv.ParseInt(s, 10, 32)
if err != nil {
return 0, false
}
return int32(n), true
}
// truncate bounds an error message built from dump data, which is
// attacker-free but can be long.
func truncate(s string) string {
const limit = 64
if len(s) <= limit {
return s
}
return s[:limit] + "..."
}
+406
View File
@@ -0,0 +1,406 @@
//go:build indexbuild
package explore
import (
"archive/tar"
"bytes"
"context"
"errors"
"strings"
"testing"
"yellowjacket/backend/database"
)
// tarOf builds an uncompressed tar of the named members, in the order
// given. Order is the point of several of these tests: the real dump's
// members are alphabetical, which is what lets one pass resolve an
// entity's credit without buffering 35M recordings.
func tarOf(t *testing.T, members ...[2]string) *tar.Reader {
t.Helper()
var buf bytes.Buffer
tw := tar.NewWriter(&buf)
for _, m := range members {
body := []byte(m[1])
if err := tw.WriteHeader(&tar.Header{
Name: "mbdump/" + m[0],
Mode: 0o644,
Size: int64(len(body)),
Typeflag: tar.TypeReg,
}); err != nil {
t.Fatalf("tar header: %v", err)
}
if _, err := tw.Write(body); err != nil {
t.Fatalf("tar write: %v", err)
}
}
if err := tw.Close(); err != nil {
t.Fatalf("tar close: %v", err)
}
return tar.NewReader(&buf)
}
func tsv(rows ...[]string) string {
var b strings.Builder
for _, r := range rows {
b.WriteString(strings.Join(r, "\t"))
b.WriteByte('\n')
}
return b.String()
}
// mustMBID is testMBID in the packed form the catalog stores.
func mustMBID(label string) uuid16 {
var u uuid16
if !parseUUID(testMBID(label), u[:]) {
panic("testMBID did not produce a UUID for " + label)
}
return u
}
// The two artists of the worked example, and the entities they credit.
var (
creditRecMBID = mustMBID("recording-1")
creditRGMBID = mustMBID("release-group-1")
)
// sampleDump is the shape verified against the 20260815 export:
// artist(id, gid, ...), artist_credit(id, name, artist_count, ...),
// artist_credit_name(credit, position, artist, name, join_phrase),
// recording/release_group(id, gid, name, artist_credit, ...).
func sampleDump(t *testing.T) *tar.Reader {
t.Helper()
return tarOf(t,
[2]string{"artist", tsv(
[]string{"11", testMBID("artist-a"), "Snoop Doggy Dogg", "Snoop Doggy Dogg"},
[]string{"22", testMBID("artist-b"), "2Pac", "2Pac"},
)},
[2]string{"artist_credit", tsv(
[]string{"900", "2Pac feat. Snoop Dogg", "2", "1", "", "0", ""},
[]string{"901", "Solo Artist", "1", "1", "", "0", ""},
)},
[2]string{"artist_credit_name", tsv(
// Deliberately out of position order: the dump is not
// obliged to emit them sorted and the credit's meaning is
// the order, not the file's.
[]string{"900", "1", "11", "Snoop Dogg", ""},
[]string{"900", "0", "22", "2Pac", " feat. "},
[]string{"901", "0", "11", "Solo Artist", ""},
)},
[2]string{"recording", tsv(
[]string{"1", testMBID("recording-1"), "Some Song", "900", "180000"},
[]string{"2", testMBID("not-kept"), "Other", "900", "1"},
[]string{"3", testMBID("solo"), "Solo", "901", "1"},
)},
[2]string{"release_group", tsv(
[]string{"5", testMBID("release-group-1"), "Some Album", "900", "1"},
)},
)
}
func creditTestImporter(t *testing.T) *dumpImporter {
t.Helper()
db := database.NewTestDB(t)
return &dumpImporter{
si: NewSearchIndex(db, nil, nil, testLogger()),
logger: testLogger(),
}
}
// TestScanCreditDumpDecomposes is the worked example end to end: the
// credit's parts come back in position order, with the *credited*
// names and the join phrase between them.
func TestScanCreditDumpDecomposes(t *testing.T) {
imp := creditTestImporter(t)
kept := map[uuid16]struct{}{
creditRecMBID: {},
creditRGMBID: {},
}
scan, err := imp.scanCreditTar(context.Background(), sampleDump(t), kept)
if err != nil {
t.Fatalf("scan: %v", err)
}
if got := len(scan.refs); got != 2 {
t.Fatalf("refs = %d, want 2 (the recording and the release group)", got)
}
if scan.refs[creditRecMBID] != 900 {
t.Errorf("recording credit = %d, want 900", scan.refs[creditRecMBID])
}
parts := scan.parts[900]
if len(parts) != 2 {
t.Fatalf("parts = %d, want 2", len(parts))
}
// Sorting happens on write, so assert the pieces are all present
// and let the render test below check the order.
byPos := map[int]creditPart{}
for _, p := range parts {
byPos[p.position] = p
}
if byPos[0].name != "2Pac" || byPos[0].join != " feat. " {
t.Errorf("position 0 = %q/%q, want \"2Pac\"/\" feat. \"",
byPos[0].name, byPos[0].join)
}
// The credited name, not the artist's own name: this is the whole
// reason credited_name is stored per row.
if byPos[1].name != "Snoop Dogg" {
t.Errorf("position 1 credited name = %q, want \"Snoop Dogg\"", byPos[1].name)
}
}
// TestSingleArtistCreditsAreNotStored: a one-artist credit is already
// described by explore_index's artist_name/artist_mbid, and storing it
// would roughly triple the table to say nothing new.
func TestSingleArtistCreditsAreNotStored(t *testing.T) {
imp := creditTestImporter(t)
solo := mustMBID("solo")
kept := map[uuid16]struct{}{solo: {}}
scan, err := imp.scanCreditTar(context.Background(), sampleDump(t), kept)
if err != nil {
t.Fatalf("scan: %v", err)
}
if len(scan.refs) != 0 {
t.Fatalf("a single-artist credit was referenced: %v", scan.refs)
}
if _, ok := scan.multiCredits[901]; ok {
t.Error("credit 901 has artist_count 1 and should not be multi")
}
}
// TestOnlyKeptEntitiesAreReferenced: the catalog's popularity filter
// decides what is worth carrying credits for, and an entity outside it
// must not produce a row pointing at nothing.
func TestOnlyKeptEntitiesAreReferenced(t *testing.T) {
imp := creditTestImporter(t)
kept := map[uuid16]struct{}{creditRecMBID: {}}
scan, err := imp.scanCreditTar(context.Background(), sampleDump(t), kept)
if err != nil {
t.Fatalf("scan: %v", err)
}
if _, ok := scan.refs[mustMBID("not-kept")]; ok {
t.Error("an entity outside the catalog was referenced")
}
if len(scan.used) != 1 {
t.Errorf("used credits = %d, want 1", len(scan.used))
}
}
// TestWriteCreditsRoundTrips checks what the frontend will actually
// read: parts in position order, dashed MBIDs out of the 16 raw bytes,
// and a rendered credit that reassembles to the tagged string.
func TestWriteCreditsRoundTrips(t *testing.T) {
imp := creditTestImporter(t)
kept := map[uuid16]struct{}{creditRecMBID: {}, creditRGMBID: {}}
scan, err := imp.scanCreditTar(context.Background(), sampleDump(t), kept)
if err != nil {
t.Fatalf("scan: %v", err)
}
if err := imp.writeCredits(context.Background(), scan); err != nil {
t.Fatalf("writeCredits: %v", err)
}
rows, err := imp.si.db.QueryContext(
`SELECT p.position, p.artist_mbid, p.credited_name, p.join_phrase
FROM artist_credit_ref r
JOIN artist_credit_part p ON p.credit_id = r.credit_id
WHERE r.mbid = ?
ORDER BY p.position`,
creditRecMBID[:],
)
if err != nil {
t.Fatalf("query: %v", err)
}
defer func() { _ = rows.Close() }()
var rendered strings.Builder
names := []string{}
for rows.Next() {
var (
pos int
mbid []byte
name string
join string
)
if err := rows.Scan(&pos, &mbid, &name, &join); err != nil {
t.Fatalf("scan row: %v", err)
}
if len(mbid) != 16 {
t.Fatalf("artist_mbid is %d bytes, want 16", len(mbid))
}
names = append(names, name)
rendered.WriteString(name)
rendered.WriteString(join)
}
if err := rows.Err(); err != nil {
t.Fatalf("rows: %v", err)
}
// Concatenation is the contract: names in order, join phrases
// between them, and no searching a name inside a credit string.
if got := rendered.String(); got != "2Pac feat. Snoop Dogg" {
t.Errorf("rendered credit = %q, want %q", got, "2Pac feat. Snoop Dogg")
}
if len(names) != 2 || names[0] != "2Pac" {
t.Errorf("parts came back out of position order: %v", names)
}
}
// TestCreditRefsNeverDangle: a ref whose parts were not stored renders
// as a credit with no artists at all, which is worse than the
// single-artist fallback it replaced.
func TestCreditRefsNeverDangle(t *testing.T) {
imp := creditTestImporter(t)
kept := map[uuid16]struct{}{creditRecMBID: {}}
scan, err := imp.scanCreditTar(context.Background(), sampleDump(t), kept)
if err != nil {
t.Fatalf("scan: %v", err)
}
// An artist the dump never named: the credit cannot be navigated to
// and must be dropped whole, taking its ref with it.
scan.artistGIDs = map[int32]uuid16{}
if err := imp.writeCredits(context.Background(), scan); err != nil {
t.Fatalf("writeCredits: %v", err)
}
var refs, parts int
if err := imp.si.db.QueryRowWriter(
"SELECT COUNT(*) FROM artist_credit_ref",
).Scan(&refs); err != nil {
t.Fatalf("count refs: %v", err)
}
if err := imp.si.db.QueryRowWriter(
"SELECT COUNT(*) FROM artist_credit_part",
).Scan(&parts); err != nil {
t.Fatalf("count parts: %v", err)
}
if refs != 0 || parts != 0 {
t.Fatalf("refs=%d parts=%d, want 0/0 when the artists are unknown", refs, parts)
}
}
// TestCreditDumpShapeIsAsserted: the dump has no header row, so a
// column that moved would be read as its neighbour and produce a
// catalog that is quietly wrong. Loud is the requirement.
func TestCreditDumpShapeIsAsserted(t *testing.T) {
imp := creditTestImporter(t)
short := tarOf(t, [2]string{"artist", tsv([]string{"11", "only-two-columns"})})
_, err := imp.scanCreditTar(context.Background(), short, map[uuid16]struct{}{})
if err == nil {
t.Fatal("a member with a non-UUID gid was accepted")
}
if !errors.Is(err, ErrDumpShape) {
t.Errorf("error = %v, want ErrDumpShape", err)
}
}
// TestUnescapeCopy covers Postgres COPY's text escaping, which reaches
// artist names routinely -- a tab or backslash in a name would
// otherwise shift every field after it.
func TestUnescapeCopy(t *testing.T) {
tests := []struct{ in, want string }{
{`plain`, `plain`},
{`\N`, ``},
{`a\tb`, "a\tb"},
{`a\nb`, "a\nb"},
{`back\\slash`, `back\slash`},
{`AC\/DC`, `AC\/DC`},
{`trailing\`, `trailing\`},
}
for _, tt := range tests {
if got := unescapeCopy(tt.in); got != tt.want {
t.Errorf("unescapeCopy(%q) = %q, want %q", tt.in, got, tt.want)
}
}
}
// TestEnsureArtistCreditsIsIdempotent pins what the index job depends
// on to decide whether to publish.
//
// The pass runs on every mode, including the `refresh` that a complete
// catalog always chooses — so it must be free when there is nothing to
// do, and it must say so. A `true` here republishes the artifact; a
// `true` on every run would republish an identical one weekly, and a
// permanent `false` would mean a catalog that never gains credits at
// all.
func TestEnsureArtistCreditsIsIdempotent(t *testing.T) {
imp := creditTestImporter(t)
// The marker is what "already done" means; with it set, the pass
// must not reach the network or report a change.
imp.si.setMeta(creditsImportDoneKey, "1")
if imp.ensureArtistCredits(context.Background()) {
t.Fatal("a second run reported new credits; the artifact would republish forever")
}
}
// TestEnsureArtistCreditsReportsFailureAsNoChange: a dump that cannot be
// reached leaves the catalog exactly as it was, and must not claim
// otherwise — publishing on it would ship an artifact with no credits
// and mark the work done.
func TestEnsureArtistCreditsReportsFailureAsNoChange(t *testing.T) {
imp := creditTestImporter(t)
imp.httpClient = newDumpHTTPClient()
imp.mbdumpBaseURL = "http://127.0.0.1:1/nonexistent/"
if imp.ensureArtistCredits(context.Background()) {
t.Fatal("an unreachable dump reported new credits")
}
if imp.si.hasMeta(creditsImportDoneKey) {
t.Error("a failed pass marked itself done; it would never retry")
}
}
+164
View File
@@ -0,0 +1,164 @@
//go:build indexbuild
package explore
import (
"context"
"database/sql"
"fmt"
"sort"
)
// writeCredits persists the scanned decompositions.
//
// Only credits some catalog entity actually points at are written: the
// dump has millions of multi-artist credits and the catalog keeps ~1.8M
// entities, so storing every credit would be most of a table nothing
// can reach.
//
// The two tables are written in one transaction, because a ref pointing
// at parts that are not there renders as a credit with no artists --
// worse than the single-artist fallback it replaced.
func (imp *dumpImporter) writeCredits(ctx context.Context, scan *creditScan) error {
tx, err := imp.si.db.BeginTx()
if err != nil {
return fmt.Errorf("credit import: begin: %w", err)
}
defer func() { _ = tx.Rollback() }()
// A rebuild replaces the previous pass wholesale. These are Cache
// tables derived entirely from the dump, so there is nothing to
// merge and a stale row is a wrong credit.
for _, table := range []string{"artist_credit_part", "artist_credit_ref"} {
if _, err := tx.ExecContext(ctx, "DELETE FROM "+table); err != nil {
return fmt.Errorf("credit import: clear %s: %w", table, err)
}
}
written, err := imp.writeCreditParts(ctx, tx, scan)
if err != nil {
return err
}
refs, err := imp.writeCreditRefs(ctx, tx, scan, written)
if err != nil {
return err
}
if err := tx.Commit(); err != nil {
return fmt.Errorf("credit import: commit: %w", err)
}
imp.logger.Info("credit import: complete",
"credits", len(written),
"refs", refs,
)
return nil
}
// writeCreditParts inserts the parts of every used credit and returns
// the set of credits that were actually stored.
//
// A credit is stored whole or not at all. If any of its artists has no
// MBID -- which should not happen, the dump being self-consistent, but
// would leave a part that cannot be navigated to -- the credit is
// dropped and the entity falls back to explore_index's single artist,
// which is a worse answer rather than a broken one.
func (imp *dumpImporter) writeCreditParts(
ctx context.Context, tx *sql.Tx, scan *creditScan,
) (map[int32]struct{}, error) {
stmt, err := tx.PrepareContext(ctx,
`INSERT INTO artist_credit_part
(credit_id, position, artist_mbid, credited_name, join_phrase)
VALUES (?, ?, ?, ?, ?)`,
)
if err != nil {
return nil, fmt.Errorf("credit import: prepare part insert: %w", err)
}
defer func() { _ = stmt.Close() }()
written := make(map[int32]struct{}, len(scan.used))
for credit := range scan.used {
parts := scan.parts[credit]
if len(parts) < 2 {
// artist_credit said more than one artist and
// artist_credit_name did not deliver them. Nothing to
// decompose, so leave the entity to its single artist.
continue
}
// Position order is the credit's meaning, and the dump is not
// obliged to emit it sorted.
sort.Slice(parts, func(i, j int) bool {
return parts[i].position < parts[j].position
})
resolved := make([][]any, 0, len(parts))
ok := true
for _, part := range parts {
gid, found := scan.artistGIDs[part.artistID]
if !found {
scan.skippedUnknownArtist++
ok = false
break
}
resolved = append(resolved, []any{
credit, part.position, gid[:], part.name, part.join,
})
}
if !ok {
continue
}
for _, args := range resolved {
if _, err := stmt.ExecContext(ctx, args...); err != nil {
return nil, fmt.Errorf("credit import: insert part: %w", err)
}
}
written[credit] = struct{}{}
}
return written, nil
}
// writeCreditRefs points each kept entity at its credit, skipping any
// whose credit was not stored so a ref never dangles.
func (imp *dumpImporter) writeCreditRefs(
ctx context.Context, tx *sql.Tx, scan *creditScan, written map[int32]struct{},
) (int, error) {
stmt, err := tx.PrepareContext(ctx,
"INSERT OR REPLACE INTO artist_credit_ref (mbid, credit_id) VALUES (?, ?)",
)
if err != nil {
return 0, fmt.Errorf("credit import: prepare ref insert: %w", err)
}
defer func() { _ = stmt.Close() }()
count := 0
for mbid, credit := range scan.refs {
if _, stored := written[credit]; !stored {
continue
}
id := mbid
if _, err := stmt.ExecContext(ctx, id[:], credit); err != nil {
return 0, fmt.Errorf("credit import: insert ref: %w", err)
}
count++
}
return count, nil
}
+8
View File
@@ -104,6 +104,7 @@ type dumpImporter struct {
canonicalBaseURL string
listensBaseURL string
mbdumpBaseURL string
// Disk safety floors (fields so tests can relax them).
minStartFreeBytes uint64
@@ -144,6 +145,7 @@ func newDumpImporter(si *SearchIndex, lb *ListenBrainzClient) (*dumpImporter, er
stagingDir: stagingDir,
canonicalBaseURL: defaultCanonicalBaseURL,
listensBaseURL: defaultListensBaseURL,
mbdumpBaseURL: defaultMBDumpBaseURL,
minStartFreeBytes: dumpMinStartFreeBytes,
abortFreeBytes: dumpAbortFreeBytes,
}, nil
@@ -171,6 +173,7 @@ func (imp *dumpImporter) run(ctx context.Context) error {
// Fast path: rows already assembled, only patch passes remain.
if state.Stage == dumpStageAssembled {
imp.si.MarkReadyIfPopulated()
imp.ensureArtistCredits(ctx)
imp.runPatchPasses(ctx)
if err := ctx.Err(); err != nil {
@@ -305,6 +308,11 @@ func (imp *dumpImporter) run(ctx context.Context) error {
imp.si.MarkReadyIfPopulated()
imp.si.refreshStatusCounts()
// Multi-artist credits, from a different dump. After the catalog,
// because it asks explore_index which entities are worth carrying
// credits for.
imp.ensureArtistCredits(ctx)
// Stage 4: API patch passes (idempotent).
imp.runPatchPasses(ctx)
+14 -1
View File
@@ -177,12 +177,25 @@ func run(o opts) error {
complete := svc.IndexImportComplete() && !errors.Is(err, errIncomplete)
// Credits are maintenance, not part of any one mode. They come from
// a different dump, they are keyed on entities the catalog already
// holds, and a catalog built before the pass existed would otherwise
// only gain them from a rebuild — which re-downloads ~205 GB to
// re-derive rows it already has. Skipped when the import is not
// complete, because there is nothing to key them against yet.
creditsAdded := false
if complete {
creditsAdded = svc.EnsureArtistCredits(context.Background())
}
// "Changed" means there is something new worth publishing, so it is
// only ever true for a finished import: a build stamps the listens
// series early, long before its rows are assembled, and reporting a
// change off that would be a lie about a half-built index.
changed := complete &&
(svc.IndexBaselineSeries() != seriesBefore || chosen != modeRefresh)
(svc.IndexBaselineSeries() != seriesBefore ||
chosen != modeRefresh ||
creditsAdded)
report(logger, svc, chosen, complete, changed)
+13 -4
View File
@@ -53,10 +53,19 @@ func TestRetireLibraryTables(t *testing.T) {
CREATE TABLE recordings (id INTEGER PRIMARY KEY, title TEXT);
`)
// The symptom, before the repair: the schema cannot be applied over
// a table whose shape has moved on.
if _, err := database.NewDB(logger); err == nil {
t.Fatal("expected the stale shape to fail to open; it did not")
// This used to assert the symptom -- that the schema cannot be
// applied over a table whose shape has moved on -- because at the
// time nothing repaired it and only this job did. The app-side
// repair (backend/database/staleshape.go) now retires a stale
// non-authored table before applySchema meets it, so opening
// succeeds and the symptom no longer reproduces from here.
//
// That does not make retireLibraryTables redundant, and the rest of
// this test is why: the app-side repair only removes what is *stale*,
// while this database wants its library half gone entirely, healthy
// or not, because nothing here scans, plays or authors.
if _, err := database.NewDB(logger); err != nil {
t.Fatalf("the app-side repair should have opened this: %v", err)
}
if err := retireLibraryTables(context.Background(), logger); err != nil {
+77
View File
@@ -207,6 +207,26 @@ func createSchema(db *sql.DB) error {
key TEXT PRIMARY KEY,
value TEXT NOT NULL
)`,
// Multi-artist credits. Shipped as their own tables rather than
// as an explore_index column because a credit is a variable
// number of ordered parts, and because credits are *shared* --
// an album's tracks by one artist reference one credit, which is
// what keeps this to a few hundred thousand rows.
//
// An importer that predates these reads an artifact without
// them; artifactHasCredits is what asks.
`CREATE TABLE core.artist_credit_part (
credit_id INTEGER NOT NULL,
position INTEGER NOT NULL,
artist_mbid BLOB NOT NULL,
credited_name TEXT NOT NULL,
join_phrase TEXT NOT NULL DEFAULT '',
PRIMARY KEY (credit_id, position)
) WITHOUT ROWID`,
`CREATE TABLE core.artist_credit_ref (
mbid BLOB NOT NULL PRIMARY KEY,
credit_id INTEGER NOT NULL
) WITHOUT ROWID`,
}
for _, stmt := range stmts {
@@ -283,6 +303,63 @@ func copyRows(db *sql.DB, artists, perArtistRGs, perArtistRecs int) error {
fmt.Printf(" %-15s %d\n", sel.label+":", n)
}
return copyCredits(db)
}
// copyCredits ships the credit decomposition for the entities that made
// it into the artifact, and only those.
//
// The refs go first and the parts follow *from* the refs, so a credit is
// carried only if something in the artifact points at it. The source
// index holds credits for every catalog entity, while the artifact is a
// windowed subset -- copying all of them would carry a large table most
// of which nothing in the artifact can reach.
//
// A source index built before the credit pass simply has no rows here,
// which is not an error: the artifact then carries the tables empty, and
// every credit falls back to its single artist exactly as before.
func copyCredits(db *sql.DB) error {
// Asked, not assumed. A source index built before the credit pass
// has no such table, and "no such table" would fail an export whose
// catalog is otherwise complete.
for _, table := range []string{"artist_credit_ref", "artist_credit_part"} {
var n int
if err := db.QueryRow(
`SELECT COUNT(*) FROM main.sqlite_master
WHERE type = 'table' AND name = ?`, table,
).Scan(&n); err != nil {
return fmt.Errorf("probe %s: %w", table, err)
}
if n == 0 {
fmt.Printf(" %-15s none in source\n", "credits:")
return nil
}
}
refs, err := insertSelect(db, `
INSERT INTO core.artist_credit_ref (mbid, credit_id)
SELECT r.mbid, r.credit_id
FROM main.artist_credit_ref r
WHERE r.mbid IN (SELECT mbid FROM core.explore_index)`)
if err != nil {
return err
}
parts, err := insertSelect(db, `
INSERT INTO core.artist_credit_part
(credit_id, position, artist_mbid, credited_name, join_phrase)
SELECT p.credit_id, p.position, p.artist_mbid, p.credited_name, p.join_phrase
FROM main.artist_credit_part p
WHERE p.credit_id IN (SELECT credit_id FROM core.artist_credit_ref)`)
if err != nil {
return err
}
fmt.Printf(" %-15s %d refs, %d parts\n", "credits:", refs, parts)
return nil
}
@@ -12,6 +12,7 @@ export {
export type {
AlbumCompleteFunc,
CreditPart,
IndexStatus,
LBSimilarArtist,
LBTopRecording,
@@ -7,6 +7,23 @@
*/
export type AlbumCompleteFunc = any;
/**
* CreditPart is one credited artist within a credit, in credit order.
*
* CreditedName is the name *as credited*, which is not the artist's own
* name: MusicBrainz credits "Snoop Dogg" on a track by the artist
* called "Snoop Doggy Dogg". Display uses it; navigation uses
* ArtistMBID. JoinPhrase is the literal connector that follows this
* part, so a credit renders by concatenation and never by searching a
* name inside a credit string.
*/
export interface CreditPart {
"position": number;
"artistMbid": string;
"creditedName": string;
"joinPhrase": string;
}
/**
* IndexStatus is the full index build status, exposed to the frontend.
*/
@@ -225,6 +225,18 @@ export function GetCandidateThumbnail(releaseMBID: string, releaseGroupMBID: str
return $Call.ByID(1946932424, releaseMBID, releaseGroupMBID);
}
/**
* GetCredits is the bound form: the frontend asks for a tracklist's
* worth of MBIDs at once rather than one per row.
*
* Batched for the reason every other per-row backend question here is:
* asking on hover or on render turns a list into N IPC round trips, and
* this one is asked about every row of every list in the app.
*/
export function GetCredits(mbids: string[] | null): $CancellablePromise<{ [_ in string]?: $models.CreditPart[] | null } | null> {
return $Call.ByID(225964099, mbids);
}
/**
* GetExploreShelves builds the page Explore shows before a query.
*
@@ -52,7 +52,8 @@ import {
} from '@utils/context-menu-controller.js';
import type { ContextMenuHost } from '@utils/context-menu-controller.js';
import { FavoritesController } from '@store/controllers/favorites-controller';
import { artistLink, exploreLinkStyles } from '../../utils/explore-link';
import { creditLink, exploreLinkStyles } from '../../utils/explore-link';
import { creditStore } from '@store/credit-store';
import {
createAlbumArtDragImage,
createDragImage,
@@ -425,8 +426,18 @@ export class CoverGrid
* Lifecycle
* ==================================================================== */
/** Unsubscribes the credit-arrival repaint. */
private creditsUnsub?: () => void;
override connectedCallback() {
super.connectedCallback();
this.creditsUnsub = creditStore.subscribe(() => {
this.requestUpdate();
// Two virtualizers when the grid is split; both draw rows.
this.renderRoot?.querySelectorAll('lit-virtualizer')
.forEach((v) => (v as unknown as { requestUpdate(): void }).requestUpdate());
});
this.restoreSortPreferences();
this.loadAlbums();
@@ -441,6 +452,8 @@ export class CoverGrid
override disconnectedCallback() {
super.disconnectedCallback();
this.creditsUnsub?.();
this.creditsUnsub = undefined;
this.removeEventListener(
'error',
@@ -1820,7 +1833,7 @@ export class CoverGrid
class="artist-name"
title="${album.ArtistName}"
>
${artistLink(album.ArtistName, album.ArtistMBID ?? '')}
${creditLink(creditStore.credits(album.MBID), album.ArtistName, album.ArtistMBID ?? '')}
</div>
</div>
</div>
@@ -20,7 +20,8 @@ type MBRelease = explore.MBRelease;
type MBTrack = explore.MBTrack;
import { exploreCache } from '../../store/explore-cache';
import { libraryStore } from '../../store/library-store';
import { artistLink, exploreLinkStyles } from '../../utils/explore-link';
import { creditLink, exploreLinkStyles } from '../../utils/explore-link';
import { creditStore } from '@store/credit-store';
import { describeError } from '../../utils/describe-error';
import { EventsOn } from '@runtime/runtime';
import { Events } from '../../events';
@@ -704,8 +705,15 @@ export class ExploreAlbumDetails extends LitElement implements ContextMenuHost {
* BrowseReleases fetch never signals readiness. */
private releasesFallbackTimer?: number;
/** Unsubscribes the credit-arrival repaint. */
private creditsUnsub?: () => void;
override connectedCallback() {
super.connectedCallback();
this.creditsUnsub = creditStore.subscribe(() => {
this.requestUpdate();
});
if (this.releaseGroupMBID || this.localAlbumId) {
void this.loadAllData();
}
@@ -760,6 +768,8 @@ export class ExploreAlbumDetails extends LitElement implements ContextMenuHost {
override disconnectedCallback() {
super.disconnectedCallback();
this.creditsUnsub?.();
this.creditsUnsub = undefined;
this.downloadUnsub?.();
this.downloadUnsub = null;
this.unsubReleasesReady?.();
@@ -2850,7 +2860,11 @@ export class ExploreAlbumDetails extends LitElement implements ContextMenuHost {
return html`
${artist
? html`<div class="album-artist">
${artistLink(artist, artistMbid)}
${creditLink(
creditStore.credits(this.releaseGroupMBID),
artist,
artistMbid,
)}
</div>`
: nothing}
${metaParts.length > 0
@@ -16,7 +16,8 @@ import { exploreCache, ARTIST_IMAGE_CACHE_LIMIT } from '../../store/explore-cach
import { queueStore } from '../../store/queue-store';
import { notificationStore } from '../../store/notification-store';
import '../notifications/inline-notice';
import { artistLink, trackLink, exploreLinkStyles } from '../../utils/explore-link';
import { creditLink, trackLink, exploreLinkStyles } from '../../utils/explore-link';
import { creditStore } from '@store/credit-store';
import { describeError } from '../../utils/describe-error';
import '@awesome.me/webawesome/dist/components/icon/icon.js';
import '../library-status-indicator/library-status-indicator.js';
@@ -774,6 +775,14 @@ export class ExploreView extends ViewLifecycleMixin(LitElement) implements Conte
}
protected override onViewActivate(): void {
// A cached primary view, so this is torn down on the way out
// rather than on disconnect — which never fires here.
this.whileActive(
creditStore.subscribe(() => {
this.requestUpdate();
}),
);
// Fetched on arrival rather than on connect: this is a cached
// primary view, created and warmed at startup, so a fetch there
// is three catalog queries every user pays for whether or not
@@ -2193,7 +2202,7 @@ export class ExploreView extends ViewLifecycleMixin(LitElement) implements Conte
<div class="album-title" title="${rg.title}">
${rg.title}
</div>
<div class="album-artist">${artistLink(rg.artistCredit, rg.artistMbid ?? '')}</div>
<div class="album-artist">${creditLink(creditStore.credits(rg.mbid), rg.artistCredit, rg.artistMbid ?? '')}</div>
<div class="album-meta">
<div class="album-meta-text">
${rg.primaryType
@@ -2255,7 +2264,7 @@ export class ExploreView extends ViewLifecycleMixin(LitElement) implements Conte
${trackLink(r.title, r.releaseName ?? '', r.releaseGroupMbid ?? '', r.mbid)}
</div>
<div class="track-artist">
${artistLink(r.artistCredit, r.artistMbid ?? '')}
${creditLink(creditStore.credits(r.mbid), r.artistCredit, r.artistMbid ?? '')}
</div>
</div>
<div class="track-meta">
@@ -5,11 +5,12 @@ import '../audio-player/controls/player-controls';
import '../audio-player/seekbar/seek-bar';
import '../audio-player/volume-control/volume-control';
import {
artistLink,
creditLink,
albumLink,
exploreLinkStyles,
} from '@utils/explore-link';
import { PlayerController } from '@store/controllers/player-controller';
import { creditStore } from '@store/credit-store';
import { FavoritesController } from '@store/controllers/favorites-controller';
import { designTokens } from '../../styles/tokens.css';
import { srOnly } from '../../styles/sr-only.css';
@@ -36,6 +37,22 @@ import { srOnly } from '../../styles/sr-only.css';
@customElement('now-playing-view')
export class NowPlayingView extends LitElement {
private player = new PlayerController(this);
/** Unsubscribes the credit-arrival repaint. */
private creditsUnsub?: () => void;
override connectedCallback(): void {
super.connectedCallback();
// Credits arrive after the track does, so the name this view is
// already showing has to be re-rendered when they land.
this.creditsUnsub = creditStore.subscribe(() => this.requestUpdate());
}
override disconnectedCallback(): void {
super.disconnectedCallback();
this.creditsUnsub?.();
this.creditsUnsub = undefined;
}
private favCtrl = new FavoritesController(this);
static override styles = [designTokens, srOnly, exploreLinkStyles, css`
@@ -262,7 +279,11 @@ export class NowPlayingView extends LitElement {
${track.title || track.fileName}
</h2>
<p class="artist">
${artistLink(track.artist, track.artistMbid)}
${creditLink(
creditStore.credits(track.recordingMbid),
track.artist,
track.artistMbid,
)}
</p>
${track.album
? html`<p class="album">
@@ -4,7 +4,7 @@ import '@awesome.me/webawesome/dist/components/icon/icon.js';
import '@awesome.me/webawesome/dist/components/popup/popup.js';
import type WaPopup from '@awesome.me/webawesome/dist/components/popup/popup.js';
import {
artistLink,
creditLink,
trackLink,
exploreLinkStyles,
} from '@utils/explore-link';
@@ -14,6 +14,7 @@ import {
navigateToQueueSource,
} from '@utils/queue-source-link';
import { PlayerController } from '@store/controllers/player-controller';
import { creditStore } from '@store/credit-store';
import { QueueController } from '@store/controllers/queue-controller';
import { FavoritesController } from '@store/controllers/favorites-controller';
import { designTokens } from '../../styles/tokens.css';
@@ -325,6 +326,9 @@ export class NowPlaying extends LitElement {
}
`];
/** Unsubscribes the credit-arrival repaint. */
private creditsUnsub?: () => void;
override connectedCallback() {
super.connectedCallback();
this.loadScrollMode();
@@ -341,10 +345,21 @@ export class NowPlaying extends LitElement {
this.geometryDirty = true;
this.requestUpdate();
});
// A credit arriving changes the rendered text, and the marquee
// measures that text — so this is a geometry change, not just a
// repaint. Saying so is what stops the bar scrolling to the
// old width.
this.creditsUnsub = creditStore.subscribe(() => {
this.geometryDirty = true;
this.requestUpdate();
});
}
override disconnectedCallback() {
super.disconnectedCallback();
this.creditsUnsub?.();
this.creditsUnsub = undefined;
// A drag interrupted by the bar going away still has to clean up.
this.attachDragListeners(false);
window.removeEventListener(SCROLL_CHANGE_EVENT, this.handleScrollModeEvent);
@@ -483,7 +498,7 @@ export class NowPlaying extends LitElement {
@mouseleave=${this.handleArtistMouseLeave}
@transitionend=${() => this.onScrollCycleEnd('artist')}
>
<span class="scroll-content">${artistLink(track.artist, track.artistMbid) || 'Unknown Artist'}</span>
<span class="scroll-content">${creditLink(creditStore.credits(track.recordingMbid), track.artist, track.artistMbid) || 'Unknown Artist'}</span>
</span>
${describeQueueSource(this.queue.source)
? html`
@@ -24,6 +24,7 @@ import type * as playlist from '@go/playlist/models.js';
import { EventsOn } from '@runtime/runtime';
import { Events } from '../../events';
import { queueStore } from '@store/queue-store';
import { creditStore } from '@store/credit-store';
import { PlayerController } from '@store/controllers/player-controller';
import { SearchController } from '@store/controllers/search-controller';
import { SelectionController } from '@utils/selection-controller';
@@ -61,7 +62,8 @@ import '@components/duplicate-tracks-dialog/duplicate-tracks-dialog.js';
import type { DuplicateTracksDialog } from '@components/duplicate-tracks-dialog/duplicate-tracks-dialog.js';
import { formatMilliseconds } from '@utils/time';
import {
artistLink,
creditLink,
creditText,
albumLink,
trackLink,
exploreLinkStyles,
@@ -106,6 +108,9 @@ export class PlaylistDetails
* rather than guessed. Without the hint the flow layout's 100 px
* default drives constant scroll-error correction, which reads as
* the list jumping under the pointer. */
/** Unsubscribes the credit-arrival repaint. */
private creditsUnsub?: () => void;
@query('lit-virtualizer')
private virtualizer?: LitVirtualizer;
@@ -224,6 +229,14 @@ export class PlaylistDetails
override connectedCallback() {
super.connectedCallback();
// Credits arrive after the rows that asked for them, and a
// virtualizer repaints from its *own* properties — a host
// update alone leaves the rows exactly as they were.
this.creditsUnsub = creditStore.subscribe(() => {
this.requestUpdate();
this.virtualizer?.requestUpdate();
});
this.loadTracks();
this.tracksChangedCleanup = EventsOn(
@@ -248,6 +261,8 @@ export class PlaylistDetails
override disconnectedCallback() {
super.disconnectedCallback();
this.creditsUnsub?.();
this.creditsUnsub = undefined;
if (this.tracksChangedCleanup) {
this.tracksChangedCleanup();
@@ -1575,7 +1590,7 @@ export class PlaylistDetails
: nothing}
</div>
<span class="cell col-title" title="${track.Title || track.FilePath}">${trackLink(track.Title, track.Album, track.ReleaseGroupMBID, track.RecordingMBID, undefined, track.Artist) || track.FilePath}</span>
<span class="cell col-artist" title="${track.Artist}">${artistLink(track.Artist, track.ArtistMBID)}</span>
<span class="cell col-artist" title="${creditText(creditStore.credits(track.RecordingMBID), track.Artist)}">${creditLink(creditStore.credits(track.RecordingMBID), track.Artist, track.ArtistMBID)}</span>
<span class="cell col-album" title="${track.Album}">${albumLink(track.Album, track.ReleaseGroupMBID, undefined, track.Artist)}</span>
<span class="cell col-duration">${formatMilliseconds(track.Duration)}</span>`}
</div>
@@ -12,6 +12,7 @@ import '@awesome.me/webawesome/dist/components/popup/popup.js';
import type WaPopup from '@awesome.me/webawesome/dist/components/popup/popup.js';
import '@awesome.me/webawesome/dist/components/dropdown-item/dropdown-item.js';
import { QueueController } from '@store/controllers/queue-controller';
import { creditStore } from '@store/credit-store';
import {
describeQueueSource,
isQueueSourceNavigable,
@@ -55,7 +56,7 @@ import { tracksByFilePath } from '@utils/track-index.js';
import type { TrackDetails } from '@components/track-details/track-details.js';
import type { CoverArtUrls } from '@components/track-details/track-details.js';
import {
artistLink,
creditLink,
trackLink,
exploreLinkStyles,
} from '@utils/explore-link';
@@ -108,6 +109,9 @@ export class QueuePanel
@query('#playlist-submenu')
private playlistSubmenuPopup!: WaPopup;
/** Unsubscribes the credit-arrival repaint. */
private creditsUnsub?: () => void;
@query('lit-virtualizer')
private virtualizer!: LitVirtualizer;
@@ -635,6 +639,14 @@ export class QueuePanel
override connectedCallback() {
super.connectedCallback();
// Credits arrive after the rows that asked for them, and a
// virtualizer repaints from its *own* properties — a host
// update alone leaves the rows exactly as they were.
this.creditsUnsub = creditStore.subscribe(() => {
this.requestUpdate();
this.virtualizer?.requestUpdate();
});
this.style.setProperty(
'--queue-width',
`${this.panelWidth}px`,
@@ -671,6 +683,8 @@ export class QueuePanel
override disconnectedCallback() {
super.disconnectedCallback();
this.creditsUnsub?.();
this.creditsUnsub = undefined;
document.removeEventListener(
'mousemove',
this.handleMouseMove,
@@ -1675,7 +1689,7 @@ export class QueuePanel
${trackLink(title, track.album, track.releaseGroupMbid, track.recordingMbid, undefined, track.artist)}
</span>
<span class="track-artist" title=${artist}>
${artistLink(track.artist, track.artistMbid) || 'Unknown Artist'}
${creditLink(creditStore.credits(track.recordingMbid), track.artist, track.artistMbid) || 'Unknown Artist'}
</span>
</div>
<button
@@ -15,6 +15,7 @@ import {
import { EventsOn } from '@runtime/runtime';
import { Events } from '../../events';
import { queueStore } from '@store/queue-store';
import { creditStore } from '@store/credit-store';
import { PlayerController } from '@store/controllers/player-controller';
import { SearchController } from '@store/controllers/search-controller';
import { SelectionController } from '@utils/selection-controller';
@@ -51,7 +52,8 @@ import type { CoverArtUrls } from '@components/track-details/track-details.js';
import { libraryStore } from '@store/library-store';
import { formatMilliseconds } from '@utils/time';
import {
artistLink,
creditLink,
creditText,
albumLink,
trackLink,
exploreLinkStyles,
@@ -136,6 +138,9 @@ export class SmartPlaylistDetails
* rather than guessed: without the hint the flow layout's 100 px
* default drives constant scroll-error correction, which reads as
* the list jumping under the pointer. */
/** Unsubscribes the credit-arrival repaint. */
private creditsUnsub?: () => void;
@query('lit-virtualizer')
private virtualizer?: LitVirtualizer;
@@ -609,6 +614,14 @@ export class SmartPlaylistDetails
override connectedCallback() {
super.connectedCallback();
// Credits arrive after the rows that asked for them, and a
// virtualizer repaints from its *own* properties — a host
// update alone leaves the rows exactly as they were.
this.creditsUnsub = creditStore.subscribe(() => {
this.requestUpdate();
this.virtualizer?.requestUpdate();
});
if (this.autoEdit) {
// Skip evaluation for new playlists — go straight to editor.
this.autoEdit = false;
@@ -649,6 +662,8 @@ export class SmartPlaylistDetails
override disconnectedCallback() {
super.disconnectedCallback();
this.creditsUnsub?.();
this.creditsUnsub = undefined;
if (this.playlistDeletedCleanup) {
this.playlistDeletedCleanup();
@@ -1423,7 +1438,7 @@ export class SmartPlaylistDetails
: nothing}
</div>
<span class="cell col-title" title="${track.Title || track.FilePath}">${trackLink(track.Title, track.Album, track.ReleaseGroupMBID, track.RecordingMBID, undefined, track.Artist) || track.FilePath}</span>
<span class="cell col-artist" title="${track.Artist}">${artistLink(track.Artist, track.ArtistMBID)}</span>
<span class="cell col-artist" title="${creditText(creditStore.credits(track.RecordingMBID), track.Artist)}">${creditLink(creditStore.credits(track.RecordingMBID), track.Artist, track.ArtistMBID)}</span>
<span class="cell col-album" title="${track.Album}">${albumLink(track.Album, track.ReleaseGroupMBID, undefined, track.Artist)}</span>
<span class="cell col-duration">${formatMilliseconds(track.Duration)}</span>`}
</div>
@@ -9,7 +9,8 @@ import {
} from '@go/explore/service.js';
import '../library-status-indicator/library-status-indicator.js';
import type { LibraryStatus } from '../library-status-indicator/library-status-indicator.js';
import { artistLink, exploreLinkStyles } from '../../utils/explore-link';
import { creditLink, exploreLinkStyles } from '../../utils/explore-link';
import { creditStore } from '@store/credit-store';
import { libraryStatusFor } from '../../utils/library-status';
import { downloadStore } from '../../store/download-store';
@@ -61,14 +62,23 @@ export class TopResultsRow extends LitElement {
* the property and never updates this element. One subscription for
* the row, not one per card.
*/
/** Unsubscribes the credit-arrival repaint. */
private creditsUnsub?: () => void;
override connectedCallback(): void {
super.connectedCallback();
this.creditsUnsub = creditStore.subscribe(() => {
this.requestUpdate();
});
this.unsubRequests = downloadStore.subscribe(() =>
this.requestUpdate(),
);
}
override disconnectedCallback(): void {
this.creditsUnsub?.();
this.creditsUnsub = undefined;
this.unsubRequests?.();
this.unsubRequests = undefined;
super.disconnectedCallback();
@@ -327,7 +337,7 @@ export class TopResultsRow extends LitElement {
${artistPart || metaPart
? html`<span class="card-subtitle"
>${artistPart
? artistLink(artistPart, r.artistMbid ?? '')
? creditLink(creditStore.credits(r.mbid), artistPart, r.artistMbid ?? '')
: nothing}${artistPart && metaPart
? ' · '
: ''}${metaPart}</span
@@ -25,6 +25,7 @@ import type { SortOption } from '@components/page-header/page-header';
import { TrackListController } from '@store/controllers/tracklist-controller';
import { FavoritesController } from '@store/controllers/favorites-controller';
import { queueStore } from '@store/queue-store';
import { creditStore } from '@store/credit-store';
import type { QueueSource } from '@store/queue-store';
import { LibraryController } from '@store/controllers/library-controller';
import {
@@ -40,6 +41,7 @@ import {
} from './search-ranking';
import {
artistLink,
creditLink,
albumLink,
trackLink,
exploreLinkStyles,
@@ -1339,6 +1341,17 @@ export class TrackList
'shortcut:tracklist-delete',
this.handleShortcutDelete,
);
// Credits arrive after the rows that asked for them. The
// virtualizer produces its rows from its *own* properties, so a
// host re-render alone repaints nothing — the same reason a
// selection change pushes requestUpdate() into it.
this.whileActive(
creditStore.subscribe(() => {
this.requestUpdate();
this.virtualizer?.requestUpdate();
}),
);
}
/**
@@ -2150,7 +2163,15 @@ export class TrackList
if (col.id === 'trackName') {
display = trackLink(track.TrackName, track.Album, track.ReleaseGroupMBID, track.RecordingMBID, display as any, track.ArtistName);
} else if (col.id === 'artistName') {
display = artistLink(track.ArtistName, track.ArtistMBID, display as any);
// A search term highlights the *flat* credit string,
// and mapping those spans onto decomposed parts is a
// different problem from rendering the credit. While
// filtering, the single link is the honest answer.
creditStore.request(track.RecordingMBID);
const parts = term ? undefined : creditStore.get(track.RecordingMBID);
display = parts && parts.length > 1
? creditLink(parts, track.ArtistName, track.ArtistMBID)
: artistLink(track.ArtistName, track.ArtistMBID, display as any);
} else if (col.id === 'album') {
display = albumLink(track.Album, track.ReleaseGroupMBID, display as any, track.ArtistName);
}
+226
View File
@@ -0,0 +1,226 @@
/**
* Multi-artist credits, keyed by recording MBID.
*
* A credit is ordered parts and the credit *string* is derived from
* them. This store holds the parts for entities that have more than
* one credited artist; everything else renders the single link it
* always did.
*
* Three things about it are load-bearing.
*
* **Absence is an answer, and it is cached as one.** The backend
* returns nothing for a single-artist credit, which is the common case
* by a wide margin — measured on a real library, 13% of tracks are
* multi-artist. Caching only the hits would re-request the other 87%
* on every render, forever, which is the same shape as the bug that
* made `explore-album-details` ask the backend on hover. A miss is
* stored as an empty array: *asked*, not *answered*.
*
* **The lookup is batched, and coalesced across callers.** Every row
* of every tracklist asks this question, and one IPC round trip per row
* is how a 5,000-row list becomes unusable. A virtualized list cannot
* hand over "the whole list" either — 50,000 rows is 100 queries for
* the ~30 on screen. So `request()` is per-row and cheap: it collects
* into a pending set and flushes once on the next frame, which turns a
* screenful of rows into exactly one call. `ensure()` remains for a
* caller that genuinely has a bounded list in hand.
*
* **It is bounded.** A cache that grows with use is a leak with a
* schedule; a browsing afternoon touches far more credits than a
* screenful. The cap is entries rather than bytes because a credit is
* a handful of short strings, unlike the art caches next door.
*/
import { GetCredits } from '@go/explore/service.js';
import type { CreditPart } from '../utils/explore-link';
import { LRUMap } from '../utils/lru-map';
import { compact } from '../utils/binding';
import { registerCacheProbe } from '../utils/cache-stats';
/**
* Entries retained. A credit is ~4 short strings, so this is well
* under a megabyte — sized to comfortably exceed any single list the
* app renders, because a cap below the visible count evicts rows that
* are still on screen and the re-render fetches them straight back.
*/
export const CREDIT_CACHE_LIMIT = 20_000;
/** An empty parts array is the negative marker: asked, no decomposition. */
type CachedParts = readonly CreditPart[];
class CreditStore {
private cache = new LRUMap<string, CachedParts>(CREDIT_CACHE_LIMIT);
/** MBIDs with a request in flight, so a re-render does not refetch. */
private inFlight = new Set<string>();
private listeners = new Set<() => void>();
/** Collected by request(), flushed as one batch on the next frame. */
private pending = new Set<string>();
private flushHandle: number | null = null;
constructor() {
registerCacheProbe('credits', () => ({
entries: this.cache.size,
chars: this.retainedChars(),
limit: CREDIT_CACHE_LIMIT,
}));
}
/**
* The strings actually retained, counted rather than estimated —
* a bound that is only checkable against a guess is not checkable.
*/
private retainedChars(): number {
let total = 0;
for (const parts of this.cache.values()) {
for (const part of parts) {
total +=
part.creditedName.length +
part.joinPhrase.length +
part.artistMbid.length;
}
}
return total;
}
/**
* Subscribe to "some credits arrived".
*
* Deliberately not per-MBID: a list fetches its rows in one call and
* re-renders once, so a fine-grained signal would buy nothing and
* cost a listener per row.
*/
subscribe(fn: () => void): () => void {
this.listeners.add(fn);
return () => this.listeners.delete(fn);
}
/**
* The parts for one entity, or undefined when it has not been asked
* about yet.
*
* An entity with a single-artist credit returns an empty array, and
* `creditLink` treats fewer than two parts as the fallback — so a
* caller does not have to distinguish "not asked" from "one artist"
* to render correctly, only to decide whether to ask.
*/
get(mbid: string | undefined): readonly CreditPart[] | undefined {
if (!mbid) return undefined;
return this.cache.get(mbid);
}
/**
* Ask about one entity, joining whatever batch is forming.
*
* Safe to call from a render: it is a set insert and a scheduled
* flush, and an entity already cached or in flight is dropped. The
* loop it looks like it might cause does not happen — after a flush
* every requested MBID is cached, so the re-render's requests are
* all dropped and nothing notifies again.
*/
request(mbid: string | undefined): void {
if (!mbid) return;
if (this.cache.has(mbid)) return;
if (this.inFlight.has(mbid)) return;
if (this.pending.has(mbid)) return;
this.pending.add(mbid);
if (this.flushHandle !== null) return;
// A frame, not a microtask: the point is to collect every row a
// virtualizer renders in this pass, and those happen across the
// whole update, not within one microtask checkpoint.
this.flushHandle = requestAnimationFrame(() => {
this.flushHandle = null;
const batch = [...this.pending];
this.pending.clear();
void this.ensure(batch);
});
}
/**
* Ask and read in one call, for use inside a template.
*
* A getter with a side effect, deliberately: the alternative is
* every call site writing `request(x)` beside `get(x)` and one of
* them eventually forgetting, which renders a permanently
* single-artist credit that looks exactly like an entity with one
* artist. Making the request the same act as the read is what
* stops the two drifting apart.
*/
credits(mbid: string | undefined): readonly CreditPart[] | undefined {
this.request(mbid);
return this.get(mbid);
}
/**
* Fetch the credits for a list, skipping anything already known or
* already being fetched.
*
* `has` rather than `get` for the membership test: probing must not
* mark an entry recently-used, or scrolling past a row would keep
* it alive ahead of one actually being rendered.
*/
async ensure(mbids: readonly (string | undefined)[]): Promise<void> {
const wanted = new Set<string>();
for (const mbid of mbids) {
if (!mbid) continue;
if (this.cache.has(mbid)) continue;
if (this.inFlight.has(mbid)) continue;
wanted.add(mbid);
}
if (wanted.size === 0) return;
const batch = [...wanted];
for (const mbid of batch) this.inFlight.add(mbid);
try {
const found = compact(await GetCredits(batch));
for (const mbid of batch) {
// Every MBID asked for gets an entry, present or not:
// the absent ones are the answer "one artist", and not
// recording that is what would re-ask forever.
this.cache.set(mbid, found[mbid] ?? []);
}
this.notify();
} catch (err) {
// A credit is an enrichment: without it every name renders
// as the single link it did before, which is a worse answer
// rather than a broken one. Nothing user-facing is worth
// interrupting for, so this stays in the console.
console.error('Failed to load artist credits', err);
} finally {
for (const mbid of batch) this.inFlight.delete(mbid);
}
}
/** Drop everything. The tags on disk changed, so credits may have. */
invalidate(): void {
this.cache = new LRUMap<string, CachedParts>(CREDIT_CACHE_LIMIT);
this.notify();
}
private notify(): void {
for (const fn of this.listeners) fn();
}
}
export const creditStore = new CreditStore();
+78
View File
@@ -294,3 +294,81 @@ async function openAlbum(
navigate(target, detail);
}
/**
* One credited artist within a multi-artist credit.
*
* Mirrors `artist_credit_part` / `file_artists`: the name **as
* credited** (which is not the artist's own name — MusicBrainz credits
* "Snoop Dogg" on a track by the artist called "Snoop Doggy Dogg"), the
* MBID to navigate to, and the literal connector that follows this
* part.
*/
export interface CreditPart {
/** The name as credited. Display uses this. */
creditedName: string;
/** The artist's MusicBrainz ID. Navigation uses this. */
artistMbid: string;
/** The connector following this part: " feat. ", " & ", ", ", "". */
joinPhrase: string;
}
/**
* Render a credit as links, one per credited artist, with the join
* phrases as plain text between them.
*
* Join phrases are **assembly instructions, not disassembly
* instructions**. This concatenates parts; it never searches for a
* name inside a credit string. That distinction is the whole point:
* the stored credit text may have come from a file's tags while the
* parts come from the catalog, and measured on a real library those
* disagree for about one in three multi-artist credits ("Skrillex
* feat. Swae Lee" tagged against "Skrillex & Swae Lee" upstream). A
* search would miss, or match the wrong span. Building from parts,
* the link boundaries are known by construction.
*
* Falls back to `artistLink(fallbackName, fallbackMbid)` — today's
* behaviour exactly — when there are no parts. That is the common
* case and not a degraded one: a single-artist credit *is* one link,
* and a file with no recording MBID or no catalog row has nothing to
* decompose. Do not try to split the fallback string; there is
* genuinely no information in it to split on.
*
* @param parts - The credit's parts in position order, if known.
* @param fallbackName - The credit as a single string.
* @param fallbackMbid - The primary artist's MBID.
*/
export function creditLink(
parts: readonly CreditPart[] | undefined,
fallbackName: string,
fallbackMbid: string,
): TemplateResult | string {
// One part is one link, so it is the fallback rather than a special
// case — and a zero-part credit reaching here would otherwise
// render as nothing at all, which is worse than the single-artist
// answer it replaced.
if (!parts || parts.length < 2) {
return artistLink(fallbackName, fallbackMbid);
}
return html`${parts.map(
(part) =>
html`${artistLink(part.creditedName, part.artistMbid)}${part.joinPhrase}`,
)}`;
}
/**
* The plain-text form of a credit, for `title=` attributes and any
* other place that needs a string rather than a template.
*
* Rendered from the same parts by the same concatenation, so the
* tooltip cannot disagree with the links beneath it.
*/
export function creditText(
parts: readonly CreditPart[] | undefined,
fallbackName: string,
): string {
if (!parts || parts.length < 2) return fallbackName;
return parts.map((p) => p.creditedName + p.joinPhrase).join('');
}
+121
View File
@@ -0,0 +1,121 @@
/**
* A track credited to more than one artist has one navigable artist in
* this app and the rest are punctuation. `creditLink` is the fix: it
* renders a credit as one link per credited artist with the join
* phrases as plain text between them.
*
* The rule these tests exist to pin is that join phrases are
* **assembly** instructions, not disassembly instructions — the credit
* is built from its parts, never found by searching a name inside a
* credit string. Measured on a real library, the stored credit text and
* the catalog's parts disagree for about one in three multi-artist
* credits, so a search would miss or match the wrong span.
*/
import { describe, expect, it } from 'vitest';
import { html, render } from 'lit';
import { creditLink, creditText, type CreditPart } from '@utils/explore-link';
const TUPAC = '11111111-1111-4111-8111-111111111111';
const SNOOP = '22222222-2222-4222-8222-222222222222';
const parts: CreditPart[] = [
{ creditedName: '2Pac', artistMbid: TUPAC, joinPhrase: ' feat. ' },
{ creditedName: 'Snoop Dogg', artistMbid: SNOOP, joinPhrase: '' },
];
function renderToEl(value: unknown): HTMLElement {
const host = document.createElement('div');
render(html`${value}`, host);
return host;
}
describe('creditLink', () => {
it('renders one link per credited artist', () => {
const el = renderToEl(creditLink(parts, '2Pac feat. Snoop Dogg', TUPAC));
const links = el.querySelectorAll('a.explore-link');
expect(links).toHaveLength(2);
expect(links[0]?.textContent).toBe('2Pac');
expect(links[1]?.textContent).toBe('Snoop Dogg');
});
it('puts the join phrase between the links as plain text', () => {
const el = renderToEl(creditLink(parts, '2Pac feat. Snoop Dogg', TUPAC));
// The whole credit reads correctly...
expect(el.textContent?.replace(/\s+/g, ' ').trim()).toBe(
'2Pac feat. Snoop Dogg',
);
// ...and " feat. " is not inside either link, which is the
// difference between a credit and a link with punctuation in it.
for (const link of el.querySelectorAll('a.explore-link')) {
expect(link.textContent).not.toMatch(/feat/);
}
});
it('falls back to a single link when there are no parts', () => {
const el = renderToEl(creditLink(undefined, 'Alina Baraz & Galimatias', TUPAC));
const links = el.querySelectorAll('a.explore-link');
expect(links).toHaveLength(1);
expect(links[0]?.textContent).toBe('Alina Baraz & Galimatias');
});
it('does not split the fallback string on its separators', () => {
// "&" and "with" appear inside real artist names — "Simon &
// Garfunkel" is one artist — so a credit with no parts is one
// link, always. This is the whole reason primaryArtist() does
// not split on them either.
const el = renderToEl(creditLink(undefined, 'Simon & Garfunkel', TUPAC));
expect(el.querySelectorAll('a.explore-link')).toHaveLength(1);
});
it('treats a one-part credit as the single-link case', () => {
// A zero- or one-part credit reaching the multi-artist branch
// would render as nothing, or as a link with a dangling join
// phrase after it.
const one: CreditPart[] = [
{ creditedName: 'Solo', artistMbid: TUPAC, joinPhrase: '' },
];
const el = renderToEl(creditLink(one, 'Solo', TUPAC));
expect(el.querySelectorAll('a.explore-link')).toHaveLength(1);
expect(el.textContent?.trim()).toBe('Solo');
});
it('renders the credited name, not the artist name', () => {
// MusicBrainz credits "Snoop Dogg" on a track by the artist
// called "Snoop Doggy Dogg". Display follows the credit;
// navigation follows the MBID.
const el = renderToEl(creditLink(parts, 'anything', TUPAC));
expect(el.textContent).toContain('Snoop Dogg');
expect(el.textContent).not.toContain('Snoop Doggy Dogg');
});
});
describe('creditText', () => {
it('reassembles the credit as a string', () => {
expect(creditText(parts, 'ignored')).toBe('2Pac feat. Snoop Dogg');
});
it('is the fallback string when there are no parts', () => {
expect(creditText(undefined, 'Alina Baraz & Galimatias')).toBe(
'Alina Baraz & Galimatias',
);
});
it('agrees with what creditLink renders', () => {
// The tooltip and the links come from the same parts by the same
// concatenation, so they cannot disagree.
const el = renderToEl(creditLink(parts, 'ignored', TUPAC));
expect(el.textContent?.replace(/\s+/g, ' ').trim()).toBe(
creditText(parts, 'ignored'),
);
});
});
+5 -1
View File
@@ -26,7 +26,11 @@ pre-commit:
go generate ./...
if [ -n "$(git diff --name-only)" ]; then
echo "Generated code is out of date. Run 'make generate' and stage the changes."
git diff --stat
# --no-pager, or this blocks forever on `less` waiting for a
# keypress that a hook run without a tty will never get: the
# commit hangs at exactly the moment it is trying to tell you
# why it failed.
git --no-pager diff --stat
exit 1
fi