Compare commits
7
Commits
d0250a2133
...
b505959934
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b505959934 | ||
|
|
409bfd5e89 | ||
|
|
0eeef6048e | ||
|
|
4fc0cdeab7 | ||
|
|
eb059a3d71 | ||
|
|
dcabec8b1d | ||
|
|
b3737d30af |
@@ -0,0 +1,337 @@
|
||||
# 015 — Multi-artist credits, navigable
|
||||
|
||||
## The problem
|
||||
|
||||
A track credited to more than one artist has exactly one navigable
|
||||
artist in this app, and the others are punctuation.
|
||||
|
||||
`audio_files` carries `artist_credit` (the credit as tagged, for
|
||||
display) and `artist_id` (one artist, for grouping and browsing).
|
||||
`primaryArtist()` (`backend/library/artistcredit.go:53`) resolves that
|
||||
one artist by *string-parsing* the credit: it strips a " feat. "
|
||||
clause, and deliberately does not split on `&`, `x`, `with` or `,`
|
||||
because those appear inside real artist names. So "Lana Del Rey ft.
|
||||
Sean Lennon" stores Lana Del Rey and discards Sean Lennon entirely,
|
||||
and "Alina Baraz & Galimatias" stores one artist whose name is the
|
||||
whole credit.
|
||||
|
||||
### What the measurement says
|
||||
|
||||
Measured 2026-08-16 against a real 26,069-file library (19,840 mp3,
|
||||
6,229 flac; 57 unreadable, m4a/ogg not examined), plus an 80+80
|
||||
MusicBrainz `inc=artist-credits` sample.
|
||||
|
||||
- **13%** of a random sample of the library's recordings have more
|
||||
than one credited artist in MusicBrainz (10 of 79 resolved).
|
||||
Extrapolates to ~3,250 of the 24,989 files carrying a recording
|
||||
MBID.
|
||||
- **0.86%** of files (224) carry any structured multi-artist signal in
|
||||
their own tags. mp3 carries **zero** files with multiple
|
||||
`MUSICBRAINZ_ARTISTID` values across 19,840 files; flac has 87.
|
||||
- **1,286** files say "feat." in `ARTIST`; **1,159 of them (90%)**
|
||||
have nothing structured behind it. A sample of 80 such files was
|
||||
multi-artist in MB **80 of 80 times**.
|
||||
|
||||
CLAUDE.md currently justifies plan 013's removal of `artist_credit` /
|
||||
`artist_credit_artist` with "3 credits of 2,823 listed more than one
|
||||
artist". That figure measured **our own writer**, not the library:
|
||||
`cachedLinkArtist` was called exactly once per credit
|
||||
(`e7748f1^:backend/library/library.go:1842`), so a collaboration could
|
||||
never have been recorded, and the three were resolution collisions on
|
||||
shared credit text. Dropping the join table was still correct — it only
|
||||
ever held one row, so it was pure join cost — but the stated evidence
|
||||
does not support "multi-artist is rare". Correcting that claim is part
|
||||
of this plan.
|
||||
|
||||
### Why the tags cannot answer it
|
||||
|
||||
Deriving the decomposition locally, with no network, works **79% of the
|
||||
time** (169 of 215 files with a multi-value `ARTISTS` tag: mp3 69/105,
|
||||
flac 100/110), and the failures are systematic rather than random:
|
||||
|
||||
```
|
||||
ARTIST = '2Pac feat. Snoop Dogg, Nate Dogg, Hussein Fatal & Yaki Kadafi'
|
||||
ARTISTS = ['2Pac', 'Snoop Doggy Dogg', 'Nate Dogg', 'Fatal', 'Yaki Kadafi']
|
||||
```
|
||||
|
||||
`ARTISTS` holds **canonical** artist names; `ARTIST` holds
|
||||
**as-credited** names. Locating one inside the other fails on
|
||||
"Snoop Doggy Dogg" vs "Snoop Dogg", on "Fatal" vs "Hussein Fatal", and
|
||||
on Unicode (`Michel'le` vs `Michel’le`, `K-Ci` vs `K‐Ci` — U+2010, not
|
||||
a hyphen). That distinction is precisely what a join phrase encodes,
|
||||
and it is why this cannot be a tag-parsing feature.
|
||||
|
||||
Two format details that will mislead anyone re-running the probe:
|
||||
Picard writes `ARTISTS` **slash-joined into one TXXX frame** on mp3 and
|
||||
as **true repeated Vorbis keys** on flac, so a probe splitting only on
|
||||
NUL undercounts mp3 to zero.
|
||||
|
||||
## The shape
|
||||
|
||||
MusicBrainz models a credit as ordered parts, and the credit *string*
|
||||
is derived from them — `artist_credit.name` is a cached render, nothing
|
||||
more. Each participant is `(position, artist, name, join_phrase)`,
|
||||
where `artist` is the MBID (canonical, what you navigate to) and `name`
|
||||
is the credited spelling (what you display).
|
||||
|
||||
**Join phrases are assembly instructions, not disassembly
|
||||
instructions.** Rendering is a concatenation, never a search:
|
||||
|
||||
```
|
||||
for each (position, artist_mbid, credited_name, join_phrase):
|
||||
emit link(credited_name -> artist_mbid)
|
||||
emit text(join_phrase)
|
||||
```
|
||||
|
||||
The link positions are known **by construction**. This is load-bearing:
|
||||
if we instead located each `credited_name` inside the stored
|
||||
`artist_credit` text, we would reintroduce the mismatch above — the
|
||||
stored string may have come from the tags while the parts come from the
|
||||
catalog, and those **disagree for ~1 in 3 multi-artist files** (61 of
|
||||
90 sampled credits rendered exactly equal to the tag string).
|
||||
Divergences seen: `'Skrillex feat. Swae Lee'` tagged vs
|
||||
`'Skrillex & Swae Lee'` in MB; `'STRFKR'` vs `'Starfucker'`;
|
||||
`'Zedd feat. Hayley Williams'` vs `'... of Paramore'`. Either MB was
|
||||
edited after tagging or Picard versions differ; either way the search
|
||||
would miss or match the wrong span.
|
||||
|
||||
So `audio_files.artist_credit` stops being the source of truth and
|
||||
becomes the **fallback**, used only where there are no parts.
|
||||
|
||||
## Where the data comes from
|
||||
|
||||
The catalog carries the decomposition; no user ever makes a
|
||||
per-recording call. Two sources were ruled out first, both cheaply:
|
||||
|
||||
- **The canonical dump — which is what CI already pulls
|
||||
(`dumpimport.go:84-85`) — does not have it.**
|
||||
`canonical_musicbrainz_data.csv` gives `artist_mbids` (ordered list)
|
||||
and `artist_credit_name`, but that last column is the *rendered*
|
||||
string. Splitting it on CI needs the as-credited names, so CI would
|
||||
fail exactly the way a local parse does.
|
||||
- **The JSON dumps do not cover the catalog.**
|
||||
`json-dumps/recording.tar.xz` is 31 MB / 368 MB uncompressed and
|
||||
holds **153,691 recordings**, not ~35M. Measured against the test
|
||||
library's 24,885 recording MBIDs: **0.00% overlap, zero rows**. It is
|
||||
some other subset and is not usable.
|
||||
|
||||
That leaves the core dump, **`mbdump.tar.bz2`** (7.1 GB compressed at
|
||||
the 20260815 export), from
|
||||
`https://data.metabrainz.org/pub/musicbrainz/data/fullexport/`. Four
|
||||
members are needed:
|
||||
|
||||
| member | why | approx rows |
|
||||
| --- | --- | --- |
|
||||
| `mbdump/artist_credit_name` | `(artist_credit, position, artist, name, join_phrase)` — the payload | ~4M |
|
||||
| `mbdump/artist` | `id -> gid`, since the above references artist *row ids* | ~2.6M |
|
||||
| `mbdump/recording` | `gid -> artist_credit`, to key credits by recording MBID | ~35M |
|
||||
| `mbdump/release_group` | same, for album credits | ~2M |
|
||||
|
||||
### Coverage is not a concern
|
||||
|
||||
Of 24,885 distinct recording MBIDs in the test library, **24,808
|
||||
(99.7%)** already have an `explore_index` recording row, measured
|
||||
against a database at 2,052,200 rows — i.e. shipped-artifact coverage,
|
||||
not a local build's. The popularity filter does not strand the long
|
||||
tail here.
|
||||
|
||||
## Status
|
||||
|
||||
- **Phase 1 — done.** `backend/explore/dumpcredits.go` +
|
||||
`dumpcreditswrite.go`, wired into `dumpimport.go`'s `run` behind its
|
||||
own `credits_import_done` marker.
|
||||
- **Phase 2 — done.** `cmd/indexexport` writes the two tables;
|
||||
`artifactimport.go` reads them behind `artifactHasCredits()`.
|
||||
- **Phase 4 — done, and it does not need Phase 3.** `explore.GetCredits`
|
||||
reads the catalog tables keyed on the *recording* MBID, which both
|
||||
sides of the app already carry — a catalog row has one and so does a
|
||||
local file (`library.Track.RecordingMBID`). So one binding serves the
|
||||
Explore pages and the library's own lists, and all ten artist-link
|
||||
call sites render credits today without a local table.
|
||||
- **Phase 3 (`file_artists`) — not started, and now an
|
||||
offline-resilience task rather than a prerequisite.** The table is
|
||||
deliberately *not* declared yet: nothing writes or reads it, and a
|
||||
schema file plus a datamap note describing behaviour that does not
|
||||
exist is a claim the code cannot back. Its remaining
|
||||
value is that credits currently vanish when the catalog is absent or
|
||||
still downloading, which is precisely the `no-index` state
|
||||
`ShelfPage.State` exists to describe. Materialising into
|
||||
`file_artists` is what makes a library stand on its own.
|
||||
|
||||
**Nothing renders yet in practice**, because no published artifact
|
||||
carries credit tables — every credit falls back to its single link
|
||||
until an index build with Phase 1 runs and is exported.
|
||||
|
||||
**Column layouts are verified against the real 20260815 export**, not
|
||||
taken from the schema docs — `artist(id, gid, …)`,
|
||||
`artist_credit(id, name, artist_count, …)`,
|
||||
`artist_credit_name(credit, position, artist, name, join_phrase)` and
|
||||
`recording(id, gid, name, artist_credit, …)` were each read out of the
|
||||
dump. `release_group` shares `recording`'s first four columns and is
|
||||
the one layout still taken on trust; `ErrDumpShape` turns a wrong guess
|
||||
into a loud failure rather than a quietly wrong catalog.
|
||||
|
||||
**Still unrun: the ingest against the real 7.1 GB dump.** Everything is
|
||||
covered by tests over a synthetic tar, which cannot catch a surprise in
|
||||
the other ~35M rows.
|
||||
|
||||
### Phase 1 — Ingest credits on CI
|
||||
|
||||
New dump stage in `cmd/indexbuild`, behind the `indexbuild` tag with
|
||||
the rest of `dumpimport.go`'s stages.
|
||||
|
||||
**Constraint from `b98840e`:** `cmd/indexbuild` is built
|
||||
`CGO_ENABLED=0` in a plain `golang` container and must not reach the
|
||||
Wails `application` package — `TestIndexToolsDoNotImportWails` walks
|
||||
`go list -deps -tags indexbuild`. Nothing here should need it, but a
|
||||
new `ServiceStartup` hook on a package this imports is how it comes
|
||||
back. Go's `compress/bzip2` is pure Go and decompress-only, which is
|
||||
all this needs.
|
||||
|
||||
**Measured, 20260815 export.** Tar members are **alphabetical**, and
|
||||
that is favourable: `artist` (435 MB), `artist_credit` (414 MB) and
|
||||
`artist_credit_name` (237 MB) all fall inside the first ~900 MB
|
||||
compressed, while `recording` and `release_group` come later. So the
|
||||
maps are complete before the rows that consume them arrive, and no
|
||||
recording data is ever buffered.
|
||||
|
||||
Pure-Go `compress/bzip2` decompresses at **26 MB/s uncompressed /
|
||||
8.7 MB/s compressed** (measured on a 250 MB prefix, 3.01x ratio) —
|
||||
**~13.7 min** for the whole file single-threaded, and less because the
|
||||
stream can stop after `release_group` rather than reading the
|
||||
`series`/`tag`/`track`/`url`/`work` tail. The 2 MB/s origin throttle
|
||||
dominates, as it already does for every other dump here.
|
||||
|
||||
Do not, however, *depend* on the ordering: assert it and fall back to
|
||||
buffering if a future export reorders, rather than silently emitting
|
||||
nothing.
|
||||
|
||||
- `artist` -> `map[int32]uuid16` (~2.6M x ~20 B = ~60 MB)
|
||||
- `artist_credit_name` -> `map[int32][]creditPart` (~4M x ~40 B =
|
||||
~200 MB)
|
||||
- `recording` / `release_group` -> emit `gid -> credit_id` **only for
|
||||
MBIDs already in `explore_index`** (the kept set is ~1.4M x 16 B =
|
||||
~22 MB), which is what keeps 35M rows from being held
|
||||
|
||||
Peak ~300 MB, one sequential pass.
|
||||
|
||||
**Only multi-artist credits are stored.** A single-artist credit is
|
||||
`(name, "")` and is already fully described by `explore_index`'s
|
||||
`artist_name` / `artist_mbid`; storing it would triple the table for
|
||||
nothing. Post-filter after loading, once the row count per credit is
|
||||
known.
|
||||
|
||||
New tables (and `datamap` entries, or `TestCatalogCoversSchema` fails
|
||||
the build — both are `Cache`, matching `explore_index`):
|
||||
|
||||
```
|
||||
artist_credit_part(credit_id, position, artist_mbid, credited_name, join_phrase)
|
||||
```
|
||||
|
||||
with `explore_index.artist_credit_id` as the link. Credits are
|
||||
**shared** — an album's twelve tracks by one artist share one credit
|
||||
row — which is the opposite of 013's local verdict, and correctly so:
|
||||
1:1 in a local library, genuinely many-to-one at 2M-row catalog scale.
|
||||
|
||||
### Phase 2 — Ship them in the artifact
|
||||
|
||||
`cmd/indexexport` currently creates exactly two tables in the artifact
|
||||
(`explore_index`, `artifact_meta`, at `cmd/indexexport/*.go:147,170`),
|
||||
so this is a structural addition, not a column.
|
||||
|
||||
Estimated size: ~13% of 1.4M recordings, deduplicated by shared credit,
|
||||
at ~2.3 parts each — order 400k rows, ~18 MB uncompressed. Against a
|
||||
~0.6 GB install that is acceptable; it must be measured rather than
|
||||
assumed before merge.
|
||||
|
||||
`artifactimport.go` must read it **only if present**, on the writer
|
||||
handle where `core` is attached — the `artifactHasTotals()` /
|
||||
`artifactStoresText()` pattern (`artifactimport.go:145-175`), one step
|
||||
up from a column to a table. An artifact published before this exists
|
||||
is still a perfectly good catalog and must import as one that declines
|
||||
to answer. Adding this to the importer's SELECT list without the probe
|
||||
is how every already-published artifact starts failing.
|
||||
|
||||
`artifactCatalogColumns` gains `artist_credit_id`; it is kept in sync
|
||||
with the exporter by `TestArtifactColumnsMatchExporter`.
|
||||
|
||||
### Phase 3 — Materialize locally
|
||||
|
||||
```
|
||||
file_artists(audio_file_id, position, artist_id, credited_name, join_phrase)
|
||||
```
|
||||
|
||||
`credited_name` is stored **per row**, not looked up from
|
||||
`artists.name` — that is the Snoop-Doggy-Dogg distinction, and it is
|
||||
the whole point.
|
||||
|
||||
Filled at scan/import time by joining `audio_files.recording_mbid`
|
||||
against the catalog. **Materialized rather than resolved live**,
|
||||
because the catalog is a downloaded artifact that can be absent or
|
||||
still arriving — that is why `ShelfPage.State` has a `no-index` value —
|
||||
and a library whose track rows lose their artists when the catalog is
|
||||
missing is worse than today.
|
||||
|
||||
That implies a backfill for the case where the catalog arrives *after*
|
||||
the library was scanned. It registers with `jobs` (progress, cancel)
|
||||
like every other long pass, and takes a **distinct kind** from
|
||||
`index-build`, since `job-controls.ts` keys its "you will discard hours
|
||||
of downloading" confirmation on that kind.
|
||||
|
||||
`artists` gains rows for guests who own no files. **This changes what
|
||||
the artists grid shows** and is an open question below.
|
||||
|
||||
### Phase 4 — Render
|
||||
|
||||
`utils/explore-link.ts` gains a credit-rendering entry point taking
|
||||
ordered parts and returning a `TemplateResult`. Every row and detail
|
||||
view already renders artist names through it, so they inherit
|
||||
multi-artist links without individually knowing credits exist — the
|
||||
property that made centralising it worthwhile.
|
||||
|
||||
Its existing fallback philosophy already covers the no-parts case: "a
|
||||
list where some rows are clickable and others silently are not reads as
|
||||
a bug, not as a statement about metadata." Where there are no parts
|
||||
(no recording MBID, or no catalog row — ~4% of the test library) render
|
||||
today's behaviour: the flat `artist_credit` string with one link to the
|
||||
primary artist. **Do not split the string there.** There is genuinely
|
||||
no information to split on, and that is the one place the temptation
|
||||
returns.
|
||||
|
||||
`primaryArtist()` stays exactly as it is. It remains the fallback and
|
||||
is still what `artist_id` means.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **Catalog credit vs tagged credit, when they disagree** (~1 in 3
|
||||
multi-artist files). Rendering the catalog's decomposition is what
|
||||
makes names navigable; preserving the file's is what makes the app
|
||||
reflect the user's files. Leaning toward: render the catalog
|
||||
decomposition, keep `artist_credit` as the fallback string. Wants a
|
||||
deliberate decision, not an accident.
|
||||
2. **Do guest artists appear in the artists grid?** Phase 3 creates
|
||||
`artists` rows for people who own no files. The grid currently means
|
||||
"artists in your library" and joins `audio_files`. A guest on one
|
||||
track is arguably in the library and arguably not. Whichever way,
|
||||
the ownership question stays "is there a file" — that rule does not
|
||||
bend.
|
||||
3. **`release_group` credits** are ingested in the same pass for
|
||||
nearly nothing, but album-artist rendering is a separate surface.
|
||||
Ship the data in phase 1, render in a follow-up rather than widening
|
||||
phase 4.
|
||||
4. **Our own `tagwriter`** does not write `ARTISTS` or multiple
|
||||
`MUSICBRAINZ_ARTISTID` frames, so autotagging a folder degrades the
|
||||
very field this rests on — the same shape as the existing
|
||||
track-totals note. Out of scope here; worth recording.
|
||||
|
||||
## Verification
|
||||
|
||||
- Coverage: re-run the library probe and assert `file_artists` is
|
||||
populated for ~13% of files, not ~0.9%.
|
||||
- `TestCatalogCoversSchema` / `TestLifetimesMatchSchema` for the new
|
||||
tables.
|
||||
- `TestIndexToolsDoNotImportWails` still passes with the new stage.
|
||||
- An artifact **without** the credits table imports cleanly (the
|
||||
`artifactHasTotals` regression shape).
|
||||
- Round-trip: a known multi-artist recording renders each name as a
|
||||
separate link with the correct join phrases between them.
|
||||
@@ -233,6 +233,51 @@ rather than renaming them.
|
||||
the drift it caused before — `sql/schemas/` and the migrations
|
||||
disagreed, and sqlc generated against the stale one.
|
||||
|
||||
**What that costs an existing database is repaired once, at open.**
|
||||
`CREATE ... IF NOT EXISTS` reaches an existing table only if its shape
|
||||
already matches and otherwise silently no-ops, so a *changed* table
|
||||
never migrates. Plan 014 added `total_tracks` to `explore_index` and
|
||||
to `indexRowFields` — the projection every explore read uses — and no
|
||||
database that already existed grew the column: **every** Explore
|
||||
search, browse, artist and album page on such an install failed with
|
||||
`no such column: total_tracks`, while a fresh install was perfectly
|
||||
healthy, which is exactly why no test saw it. Plan 013 was worse on
|
||||
the same install: `applySchema` could not be applied at all over a
|
||||
pre-013 `audio_files`, so the app did not open.
|
||||
|
||||
`backend/database/staleshape.go` runs before `applySchema` and
|
||||
retires what is stale, so the create is a create. Five things about
|
||||
it are load-bearing:
|
||||
- **It parses `sql/schemas/` for the expectation** rather than
|
||||
writing the column list down a second time, because a second list
|
||||
is a second thing to forget — the fault it exists to repair.
|
||||
- **It notices a changed *type*, not just a missing column.** 013
|
||||
moved `mbid` from TEXT to BLOB, and SQLite does not coerce between
|
||||
them: a comparison against 16 raw bytes returns no rows rather than
|
||||
an error. `ALTER TABLE ADD COLUMN` would have handled
|
||||
`total_tracks` alone and cannot express this at all, which is why
|
||||
the repair drops rather than migrates.
|
||||
- **`Authored` is never retired**, and that boundary is a test
|
||||
(`TestAuthoredTablesAreNeverRetired`), not a comment. Everything
|
||||
else is rebuildable: `Cache` by definition, `Owned` by a rescan —
|
||||
plan 013's stated "delete and rescan" — and `Derived` from Owned.
|
||||
A table the schema no longer describes at all goes too; 013 left
|
||||
seven behind plus `schema_migrations`.
|
||||
- **The drops are one transaction with `defer_foreign_keys`.** Those
|
||||
legacy tables reference each other, so dropping them in any order
|
||||
fails on whichever goes first, and turning foreign keys *off*
|
||||
instead would silently take `playlist_tracks.audio_file_id`'s
|
||||
ON DELETE SET NULL with it — leaving playlist entries pointing at
|
||||
ids a rescan reissues to *different songs*. Nulled entries are
|
||||
empty; stale ones are wrong, and wrong quietly.
|
||||
- **The order is sorted, so a failure reproduces.** Map order is
|
||||
random, and the foreign-key bug above passed its own regression
|
||||
test on two runs in three until the order was fixed.
|
||||
|
||||
Retiring `explore_index` takes its FTS and its meta with it, because
|
||||
the `dump_import_done` marker is what would otherwise stop the
|
||||
artifact ever being fetched again.
|
||||
|
||||
**What that costs an existing database is that it does not open**, and
|
||||
"delete and rescan" is the answer (plan 013, open question 1) — free
|
||||
for everyone except one machine. The index job's `/cache` volume is a
|
||||
|
||||
@@ -89,6 +89,14 @@ func NewDB(logger *slog.Logger) (*DB, error) {
|
||||
return nil, fmt.Errorf("could not apply PRAGMAs: %w", err)
|
||||
}
|
||||
|
||||
// Before the schema is applied, not after: applySchema is
|
||||
// CREATE ... IF NOT EXISTS, which no-ops against a table that
|
||||
// already exists in an older shape. Retiring the stale one first is
|
||||
// what turns that no-op into a create.
|
||||
if err := retireStaleTables(dbCtx, db, logger); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if err := applySchema(dbCtx, db); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
-- The decomposition of a multi-artist credit, from the MusicBrainz
|
||||
-- dump. One row per credited artist, in credit order.
|
||||
--
|
||||
-- A credit is ordered parts, and the credit *string* is derived from
|
||||
-- them -- MusicBrainz's own `artist_credit.name` is a cached render and
|
||||
-- nothing more. Rendering is a concatenation:
|
||||
--
|
||||
-- for each part in position order:
|
||||
-- emit link(credited_name -> artist_mbid)
|
||||
-- emit text(join_phrase)
|
||||
--
|
||||
-- so the link boundaries are known by construction. That is the whole
|
||||
-- reason this table exists, and it is why nothing may reconstruct a
|
||||
-- credit by *searching* for a name inside a credit string: the stored
|
||||
-- string may have come from a file's tags while the parts come from the
|
||||
-- catalog, and measured on a real library those disagree for about one
|
||||
-- in three multi-artist credits ("Skrillex feat. Swae Lee" tagged
|
||||
-- against "Skrillex & Swae Lee" upstream). A search would miss, or
|
||||
-- match the wrong span.
|
||||
--
|
||||
-- `credited_name` is the name *as credited*, which is not the artist's
|
||||
-- canonical name: MusicBrainz credits "Snoop Dogg" on a track by the
|
||||
-- artist whose name is "Snoop Doggy Dogg". It is stored per row rather
|
||||
-- than joined from an artist table for exactly that reason.
|
||||
--
|
||||
-- Only *multi-artist* credits are stored. A single-artist credit is
|
||||
-- (name, "") and is already fully described by explore_index's
|
||||
-- artist_name and artist_mbid; storing those would roughly triple the
|
||||
-- table to say nothing new.
|
||||
--
|
||||
-- Credits are shared: an album's twelve tracks by one artist reference
|
||||
-- one credit_id. That is the opposite of the local library's verdict
|
||||
-- in plan 013, and correctly so -- credit sharing is 1:1 in one
|
||||
-- person's files and genuinely many-to-one across a 2M-row catalog.
|
||||
--
|
||||
-- MBIDs are the same 16 raw bytes explore_index stores, for the same
|
||||
-- size reason and with the same CHECK, so a stringly write fails at the
|
||||
-- insert that made it rather than reading back as no rows at all. See
|
||||
-- backend/explore/mbid.go.
|
||||
CREATE TABLE IF NOT EXISTS artist_credit_part (
|
||||
credit_id INTEGER NOT NULL,
|
||||
position INTEGER NOT NULL,
|
||||
artist_mbid BLOB NOT NULL CHECK(length(artist_mbid) = 16),
|
||||
|
||||
-- The name as credited on this release, which may differ from the
|
||||
-- artist's canonical name. Display uses this; navigation uses the
|
||||
-- MBID above.
|
||||
credited_name TEXT NOT NULL,
|
||||
|
||||
-- The literal connector that follows this part -- " feat. ", " & ",
|
||||
-- ", ", or "" on the last part. Rendered as plain text between two
|
||||
-- links.
|
||||
join_phrase TEXT NOT NULL DEFAULT '',
|
||||
|
||||
PRIMARY KEY (credit_id, position)
|
||||
) WITHOUT ROWID;
|
||||
@@ -0,0 +1,30 @@
|
||||
-- Which credit a catalog entity is credited to. One row per recording
|
||||
-- or release group whose credit names more than one artist.
|
||||
--
|
||||
-- This is a table rather than an `explore_index.artist_credit_id`
|
||||
-- column, and that is a deliberate consequence of how this app applies
|
||||
-- its schema. `applySchema` is CREATE ... IF NOT EXISTS and there is
|
||||
-- no migration chain (plan 013), so a *column* added to an existing
|
||||
-- table never reaches a database that already has it -- while a new
|
||||
-- *table* is created on every install, old or new, for free.
|
||||
-- explore_index is the one table nobody can afford to drop and rebuild
|
||||
-- on a schema change: it is the artifact users download rather than
|
||||
-- derive.
|
||||
--
|
||||
-- Only multi-artist credits are referenced here, matching
|
||||
-- artist_credit_part. An entity with no row is credited to exactly one
|
||||
-- artist, which explore_index's own artist_name and artist_mbid already
|
||||
-- describe -- so absence is the common case and means "nothing to
|
||||
-- decompose", not "unknown".
|
||||
--
|
||||
-- `credit_id` is opaque and is only meaningful against the
|
||||
-- artist_credit_part rows built or imported alongside it. The two are
|
||||
-- always written together; nothing persists a credit_id anywhere else.
|
||||
-- The local library stores resolved parts, never this id.
|
||||
CREATE TABLE IF NOT EXISTS artist_credit_ref (
|
||||
mbid BLOB NOT NULL PRIMARY KEY CHECK(length(mbid) = 16),
|
||||
credit_id INTEGER NOT NULL
|
||||
) WITHOUT ROWID;
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_artist_credit_ref_credit
|
||||
ON artist_credit_ref(credit_id);
|
||||
@@ -27,6 +27,19 @@ type Artist struct {
|
||||
Mbid sql.NullString
|
||||
}
|
||||
|
||||
type ArtistCreditPart struct {
|
||||
CreditID int64
|
||||
Position int64
|
||||
ArtistMbid []byte
|
||||
CreditedName string
|
||||
JoinPhrase string
|
||||
}
|
||||
|
||||
type ArtistCreditRef struct {
|
||||
Mbid []byte
|
||||
CreditID int64
|
||||
}
|
||||
|
||||
type ArtistEnrichment struct {
|
||||
ArtistMbid string
|
||||
BrowsedAt sql.NullTime
|
||||
|
||||
@@ -0,0 +1,552 @@
|
||||
package database
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"log/slog"
|
||||
"maps"
|
||||
"path"
|
||||
"slices"
|
||||
"strings"
|
||||
|
||||
"yellowjacket/backend/datamap"
|
||||
)
|
||||
|
||||
// This file repairs the one thing `CREATE TABLE IF NOT EXISTS` cannot.
|
||||
//
|
||||
// `sql/schemas/` is the single description of the schema and there is no
|
||||
// migration chain (plan 013): a schema change is one edit to one file.
|
||||
// That works perfectly for a *new* table, which every install then
|
||||
// creates, and not at all for a changed one -- `IF NOT EXISTS` reaches
|
||||
// an existing table only if its shape already matches, and otherwise
|
||||
// silently no-ops. The user's answer to that is "delete and rescan"
|
||||
// (plan 013, open question 1), which is free for everything a rescan
|
||||
// rebuilds.
|
||||
//
|
||||
// It is not free for the catalog. explore_index is a *downloaded
|
||||
// artifact*, not something derived from the user's files, and it is the
|
||||
// largest thing this app stores. So it went stale instead: plan 014
|
||||
// added `total_tracks` to the schema and to `indexRowFields` -- the one
|
||||
// projection every explore read uses -- and no database that already
|
||||
// existed ever grew the column. Every Explore search, browse, artist
|
||||
// page and album page on such an install fails with
|
||||
// "no such column: total_tracks", while a fresh install is perfectly
|
||||
// healthy, which is why the tests did not see it. The same databases
|
||||
// are stale a second way, from the same plan: their `mbid` columns are
|
||||
// still TEXT where the schema now declares BLOB, and SQLite does not
|
||||
// coerce between the two -- a comparison against 16 raw bytes simply
|
||||
// returns no rows.
|
||||
//
|
||||
// The repair is to notice and drop, not to migrate. A dropped catalog
|
||||
// costs one artifact download (about a minute); the alternative --
|
||||
// ALTER TABLE ADD COLUMN, which would handle `total_tracks` alone
|
||||
// cheaply -- cannot express the TEXT-to-BLOB half at all, and would
|
||||
// leave those installs quietly broken while reporting success.
|
||||
//
|
||||
// Everything except `Authored` is eligible. `Cache` is rebuildable by
|
||||
// definition; `Owned` is a projection of the user's files and a rescan
|
||||
// rebuilds it, which is plan 013's stated answer to exactly this
|
||||
// situation ("delete and rescan", open question 1); `Derived` is
|
||||
// computed from Owned. No `Authored` table is ever dropped here --
|
||||
// that is the whole point of the datamap, and it is asserted by
|
||||
// TestAuthoredTablesAreNeverRetired rather than only stated.
|
||||
//
|
||||
// What that does *not* buy is immunity for authored rows that reference
|
||||
// a retired table. `audio_files` is MIXED KIND: `play_count`,
|
||||
// `last_played` and `tag_status` are authored columns on an Owned
|
||||
// table, and they go with it. Playlists survive as playlists, and
|
||||
// their entries survive pointing at nothing. That cost was weighed and
|
||||
// accepted rather than overlooked -- the alternative is to carry the
|
||||
// authored columns across the rebuild keyed on file_path, which stays a
|
||||
// real option if this ever bites harder than it is worth.
|
||||
//
|
||||
// **This relies on foreign_keys being ON**, which applyPRAGMAs has
|
||||
// already done by the time NewDB calls it, and the dependency is not
|
||||
// cosmetic. SQLite performs an implicit DELETE before dropping a table
|
||||
// when foreign keys are enabled, so `playlist_tracks.audio_file_id` --
|
||||
// declared ON DELETE SET NULL -- is nulled. With foreign keys off, no
|
||||
// action fires and those rows keep the ids they had, which a rescan
|
||||
// then reissues starting from 1: every playlist would silently fill
|
||||
// with *different songs*. Nulled entries are merely empty; stale ones
|
||||
// are wrong, and wrong quietly. TestRetiringOwnedTablesDoesNotDangle
|
||||
// is what stops a future reordering turning one into the other.
|
||||
|
||||
// retireGroups are tables that must be retired together. A catalog
|
||||
// whose rows are gone must not keep the full-text index built over
|
||||
// them, nor the metadata claiming the import that produced them
|
||||
// finished -- that marker is exactly what stops the artifact being
|
||||
// fetched again. applySchema recreates all three empty immediately
|
||||
// afterwards, and the ordinary "no index yet" path takes over.
|
||||
var retireGroups = [][]string{
|
||||
{
|
||||
"explore_index",
|
||||
"explore_index_fts",
|
||||
"explore_index_meta",
|
||||
"explore_champion_fts",
|
||||
},
|
||||
}
|
||||
|
||||
// schemaColumn is one column as the schema file declares it.
|
||||
type schemaColumn struct {
|
||||
name string
|
||||
typ string
|
||||
}
|
||||
|
||||
// retireStaleTables drops every non-authored table whose live shape no
|
||||
// longer matches what sql/schemas/ declares, plus any table the schema
|
||||
// no longer describes at all, so applySchema can create the current
|
||||
// shape afresh. It runs before applySchema and is a no-op on a new
|
||||
// database, where the tables do not exist yet.
|
||||
func retireStaleTables(
|
||||
ctx context.Context, db *sql.DB, logger *slog.Logger,
|
||||
) error {
|
||||
declared, err := declaredTables()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
stale := make(map[string]string)
|
||||
|
||||
for table, columns := range declared {
|
||||
entry, ok := datamap.Lookup(table)
|
||||
if !ok || entry.Kind == datamap.Authored || entry.FTS {
|
||||
continue
|
||||
}
|
||||
|
||||
reason, err := staleReason(ctx, db, table, columns)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if reason != "" {
|
||||
stale[table] = reason
|
||||
}
|
||||
}
|
||||
|
||||
obsolete, err := obsoleteTables(ctx, db)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
maps.Copy(stale, obsolete)
|
||||
|
||||
if len(stale) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
return retireGroupsFor(ctx, db, logger, stale)
|
||||
}
|
||||
|
||||
// obsoleteTables are live tables the schema no longer describes at all.
|
||||
// TestCatalogCoversSchema makes the datamap a complete description of
|
||||
// the current schema, so a table it does not know is one a past version
|
||||
// created and this one does not -- plan 013 alone left seven behind
|
||||
// (recordings, release_groups, artist_credit, artist_credit_artist,
|
||||
// release_group_recordings, recording_genres) plus the
|
||||
// schema_migrations table that squashing the chain retired. They are
|
||||
// dead weight, and one of them holding a foreign key into a table being
|
||||
// rebuilt is worse than dead weight.
|
||||
//
|
||||
// SQLite's own bookkeeping and FTS shadow tables are not obsolete:
|
||||
// datamap.Lookup resolves a shadow table to its parent, and IsInternal
|
||||
// covers the rest.
|
||||
func obsoleteTables(ctx context.Context, db *sql.DB) (map[string]string, error) {
|
||||
rows, err := db.QueryContext(
|
||||
ctx, "SELECT name FROM sqlite_master WHERE type = 'table'",
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("could not list tables: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
out := make(map[string]string)
|
||||
|
||||
for rows.Next() {
|
||||
var name string
|
||||
|
||||
if err := rows.Scan(&name); err != nil {
|
||||
return nil, fmt.Errorf("could not scan table name: %w", err)
|
||||
}
|
||||
|
||||
if datamap.IsInternal(name) {
|
||||
continue
|
||||
}
|
||||
|
||||
if _, known := datamap.Lookup(name); !known {
|
||||
out[name] = "the schema no longer describes this table"
|
||||
}
|
||||
}
|
||||
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, fmt.Errorf("could not read table list: %w", err)
|
||||
}
|
||||
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// retireGroupsFor drops each stale table along with everything its
|
||||
// retire group says must go with it.
|
||||
func retireGroupsFor(
|
||||
ctx context.Context, db *sql.DB, logger *slog.Logger,
|
||||
stale map[string]string,
|
||||
) error {
|
||||
drop := make(map[string]string)
|
||||
|
||||
for table, reason := range stale {
|
||||
drop[table] = reason
|
||||
|
||||
for _, group := range retireGroups {
|
||||
if !slices.Contains(group, table) {
|
||||
continue
|
||||
}
|
||||
|
||||
for _, member := range group {
|
||||
if _, already := drop[member]; !already {
|
||||
drop[member] = "retired with " + table
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return dropDeferred(ctx, db, logger, drop)
|
||||
}
|
||||
|
||||
// dropDeferred drops every named table in one transaction with foreign
|
||||
// key enforcement deferred to the commit.
|
||||
//
|
||||
// The deferral is required and the two obvious alternatives are both
|
||||
// wrong. These tables reference each other -- pre-013 `audio_files`
|
||||
// has a foreign key into `recordings`, which is itself being retired --
|
||||
// so dropping them one at a time in an arbitrary order fails with
|
||||
// "FOREIGN KEY constraint failed" on whichever is unlucky enough to go
|
||||
// first, and there is no order that is safe in general. Turning
|
||||
// foreign keys *off* for the duration would fix that and silently take
|
||||
// the ON DELETE SET NULL on `playlist_tracks.audio_file_id` with it,
|
||||
// leaving playlist entries pointing at ids a rescan reissues to
|
||||
// different songs -- the exact failure
|
||||
// TestRetiringOwnedTablesDoesNotDangle exists to prevent.
|
||||
//
|
||||
// Deferring keeps the actions firing while tolerating the inconsistency
|
||||
// in the middle, and the commit then checks that the end state is
|
||||
// sound. It is set inside the transaction because SQLite resets it at
|
||||
// every commit.
|
||||
func dropDeferred(
|
||||
ctx context.Context, db *sql.DB, logger *slog.Logger,
|
||||
drop map[string]string,
|
||||
) error {
|
||||
tx, err := db.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("could not begin the retire transaction: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = tx.Rollback() }()
|
||||
|
||||
if _, err := tx.ExecContext(ctx, "PRAGMA defer_foreign_keys = ON"); err != nil {
|
||||
return fmt.Errorf("could not defer foreign keys: %w", err)
|
||||
}
|
||||
|
||||
// Sorted, so a failure is reproducible. Map order is random, and a
|
||||
// bug that depends on which table happens to go first reproduces on
|
||||
// one run in three and passes review on the other two -- which is
|
||||
// exactly how the foreign-key ordering above reached a real
|
||||
// database. Sorting does not make any order *safe*; the deferral
|
||||
// does that.
|
||||
for _, table := range slices.Sorted(maps.Keys(drop)) {
|
||||
logger.Warn(
|
||||
"retiring a table the schema no longer describes",
|
||||
"table", table,
|
||||
"reason", drop[table],
|
||||
)
|
||||
|
||||
if _, err := tx.ExecContext(
|
||||
ctx, "DROP TABLE IF EXISTS "+quoteIdent(table),
|
||||
); err != nil {
|
||||
return fmt.Errorf("could not retire stale table %s: %w", table, err)
|
||||
}
|
||||
}
|
||||
|
||||
if err := tx.Commit(); err != nil {
|
||||
return fmt.Errorf("could not commit the retire: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// staleReason reports why a live table disagrees with its declaration,
|
||||
// or "" when it agrees. A column the live table does not have is the
|
||||
// additive case; a column whose declared type changed is the one an
|
||||
// ALTER could not fix anyway. Columns the live table has and the
|
||||
// schema no longer declares are ignored: they cost nothing and dropping
|
||||
// the table over one would retire a healthy catalog.
|
||||
func staleReason(
|
||||
ctx context.Context, db *sql.DB, table string, columns []schemaColumn,
|
||||
) (string, error) {
|
||||
live, err := liveColumns(ctx, db, table)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
if len(live) == 0 {
|
||||
// Not present at all: applySchema is about to create it.
|
||||
return "", nil
|
||||
}
|
||||
|
||||
for _, col := range columns {
|
||||
liveType, present := live[col.name]
|
||||
if !present {
|
||||
return "missing column " + col.name, nil
|
||||
}
|
||||
|
||||
if !sameDeclaredType(col.typ, liveType) {
|
||||
return fmt.Sprintf(
|
||||
"column %s is %s, schema declares %s",
|
||||
col.name, liveType, col.typ,
|
||||
), nil
|
||||
}
|
||||
}
|
||||
|
||||
return "", nil
|
||||
}
|
||||
|
||||
// liveColumns returns the live table's columns and their declared types,
|
||||
// empty when the table does not exist.
|
||||
func liveColumns(
|
||||
ctx context.Context, db *sql.DB, table string,
|
||||
) (map[string]string, error) {
|
||||
rows, err := db.QueryContext(
|
||||
ctx, "SELECT name, type FROM pragma_table_info(?)", table,
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("could not inspect table %s: %w", table, err)
|
||||
}
|
||||
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
out := make(map[string]string)
|
||||
|
||||
for rows.Next() {
|
||||
var name, typ string
|
||||
|
||||
if err := rows.Scan(&name, &typ); err != nil {
|
||||
return nil, fmt.Errorf("could not scan column of %s: %w", table, err)
|
||||
}
|
||||
|
||||
out[name] = typ
|
||||
}
|
||||
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, fmt.Errorf("could not read columns of %s: %w", table, err)
|
||||
}
|
||||
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// sameDeclaredType compares two SQLite type names. They are compared
|
||||
// case-insensitively and only on the leading word, so INTEGER matches
|
||||
// INTEGER and VARCHAR(20) matches VARCHAR -- SQLite's affinity rules
|
||||
// make finer distinctions meaningless, and a difference that fine is
|
||||
// not worth retiring a catalog over. An empty declared type matches
|
||||
// anything, which is what a column declared with only constraints has.
|
||||
func sameDeclaredType(declared, live string) bool {
|
||||
d := strings.ToUpper(strings.Fields(declared + " ")[0])
|
||||
l := strings.ToUpper(strings.Fields(live + " ")[0])
|
||||
|
||||
if d == "" || l == "" {
|
||||
return true
|
||||
}
|
||||
|
||||
if i := strings.IndexByte(d, '('); i >= 0 {
|
||||
d = d[:i]
|
||||
}
|
||||
|
||||
if i := strings.IndexByte(l, '('); i >= 0 {
|
||||
l = l[:i]
|
||||
}
|
||||
|
||||
return d == l
|
||||
}
|
||||
|
||||
// declaredTables parses every CREATE TABLE in sql/schemas/ into its
|
||||
// column list. Parsing the schema rather than writing the expectation
|
||||
// down a second time is the point: a second list is a second thing to
|
||||
// forget, which is the fault this whole file exists to repair.
|
||||
func declaredTables() (map[string][]schemaColumn, error) {
|
||||
dirEntries, err := schemas.ReadDir("sql/schemas")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("could not read schemas directory: %w", err)
|
||||
}
|
||||
|
||||
out := make(map[string][]schemaColumn)
|
||||
|
||||
for _, dirEntry := range dirEntries {
|
||||
if dirEntry.IsDir() {
|
||||
continue
|
||||
}
|
||||
|
||||
content, err := fs.ReadFile(schemas, path.Join("sql/schemas", dirEntry.Name()))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("could not read %s: %w", dirEntry.Name(), err)
|
||||
}
|
||||
|
||||
maps.Copy(out, parseCreateTables(string(content)))
|
||||
}
|
||||
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// constraintKeywords begin a table constraint rather than a column.
|
||||
var constraintKeywords = map[string]bool{
|
||||
"PRIMARY": true, "FOREIGN": true, "UNIQUE": true,
|
||||
"CHECK": true, "CONSTRAINT": true,
|
||||
}
|
||||
|
||||
// parseCreateTables extracts the column names and declared types of
|
||||
// every non-virtual CREATE TABLE in one schema file.
|
||||
func parseCreateTables(content string) map[string][]schemaColumn {
|
||||
out := make(map[string][]schemaColumn)
|
||||
rest := stripLineComments(content)
|
||||
|
||||
for {
|
||||
idx := indexFold(rest, "CREATE TABLE ")
|
||||
if idx < 0 {
|
||||
return out
|
||||
}
|
||||
|
||||
rest = rest[idx+len("CREATE TABLE "):]
|
||||
|
||||
head, body, ok := splitTableBody(rest)
|
||||
if !ok {
|
||||
return out
|
||||
}
|
||||
|
||||
if name := tableName(head); name != "" {
|
||||
out[name] = parseColumns(body)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// tableName pulls the table name out of the text between "CREATE TABLE"
|
||||
// and its opening parenthesis, dropping an IF NOT EXISTS and any
|
||||
// quoting.
|
||||
func tableName(head string) string {
|
||||
head = strings.TrimSpace(head)
|
||||
head = strings.TrimPrefix(head, "IF NOT EXISTS ")
|
||||
head = strings.TrimPrefix(head, "if not exists ")
|
||||
|
||||
fields := strings.Fields(head)
|
||||
if len(fields) == 0 {
|
||||
return ""
|
||||
}
|
||||
|
||||
return strings.Trim(fields[len(fields)-1], `"'`+"`")
|
||||
}
|
||||
|
||||
// splitTableBody returns the text before the table's opening paren and
|
||||
// the balanced text inside it.
|
||||
func splitTableBody(s string) (head, body string, ok bool) {
|
||||
open := strings.IndexByte(s, '(')
|
||||
if open < 0 {
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
depth := 0
|
||||
|
||||
for i := open; i < len(s); i++ {
|
||||
switch s[i] {
|
||||
case '(':
|
||||
depth++
|
||||
case ')':
|
||||
depth--
|
||||
|
||||
if depth == 0 {
|
||||
return s[:open], s[open+1 : i], true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
// parseColumns splits a table body on its top-level commas and keeps
|
||||
// the parts that are columns rather than table constraints.
|
||||
func parseColumns(body string) []schemaColumn {
|
||||
var (
|
||||
out []schemaColumn
|
||||
depth int
|
||||
start int
|
||||
)
|
||||
|
||||
parts := make([]string, 0, 8)
|
||||
|
||||
for i := range len(body) {
|
||||
switch body[i] {
|
||||
case '(':
|
||||
depth++
|
||||
case ')':
|
||||
depth--
|
||||
case ',':
|
||||
if depth == 0 {
|
||||
parts = append(parts, body[start:i])
|
||||
start = i + 1
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
parts = append(parts, body[start:])
|
||||
|
||||
for _, part := range parts {
|
||||
fields := strings.Fields(part)
|
||||
if len(fields) == 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
// A table constraint need not be followed by a space --
|
||||
// "UNIQUE(mbid)" is one field, and reading it as a column name
|
||||
// makes an entirely healthy table look stale, which retires a
|
||||
// catalog nobody asked to lose.
|
||||
head := fields[0]
|
||||
if i := strings.IndexByte(head, '('); i >= 0 {
|
||||
head = head[:i]
|
||||
}
|
||||
|
||||
if constraintKeywords[strings.ToUpper(head)] {
|
||||
continue
|
||||
}
|
||||
|
||||
col := schemaColumn{name: strings.Trim(head, `"'`+"`")}
|
||||
if len(fields) > 1 {
|
||||
col.typ = fields[1]
|
||||
}
|
||||
|
||||
out = append(out, col)
|
||||
}
|
||||
|
||||
return out
|
||||
}
|
||||
|
||||
// stripLineComments removes -- comments, which otherwise contribute
|
||||
// stray parentheses and commas to the parse.
|
||||
func stripLineComments(s string) string {
|
||||
lines := strings.Split(s, "\n")
|
||||
for i, line := range lines {
|
||||
if idx := strings.Index(line, "--"); idx >= 0 {
|
||||
lines[i] = line[:idx]
|
||||
}
|
||||
}
|
||||
|
||||
return strings.Join(lines, "\n")
|
||||
}
|
||||
|
||||
// indexFold is a case-insensitive strings.Index.
|
||||
func indexFold(s, substr string) int {
|
||||
return strings.Index(strings.ToUpper(s), strings.ToUpper(substr))
|
||||
}
|
||||
|
||||
// quoteIdent quotes a table name for interpolation into DDL, which
|
||||
// cannot take a bound parameter.
|
||||
func quoteIdent(name string) string {
|
||||
return `"` + strings.ReplaceAll(name, `"`, `""`) + `"`
|
||||
}
|
||||
@@ -0,0 +1,499 @@
|
||||
package database
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"log/slog"
|
||||
"path"
|
||||
"testing"
|
||||
|
||||
_ "modernc.org/sqlite"
|
||||
)
|
||||
|
||||
// testLogger discards the repair's warnings; the tests assert on the
|
||||
// database, not on the log.
|
||||
func testLogger() *slog.Logger {
|
||||
return slog.New(slog.DiscardHandler)
|
||||
}
|
||||
|
||||
// openRaw opens a scratch database file with no schema applied, so a
|
||||
// test can build an *old* shape and then let NewDB's repair meet it.
|
||||
func openRaw(t *testing.T, dir string) *sql.DB {
|
||||
t.Helper()
|
||||
|
||||
db, err := sql.Open("sqlite", path.Join(dir, "yj.db"))
|
||||
if err != nil {
|
||||
t.Fatalf("open: %v", err)
|
||||
}
|
||||
|
||||
t.Cleanup(func() { _ = db.Close() })
|
||||
|
||||
return db
|
||||
}
|
||||
|
||||
// TestRetiresIndexMissingAColumn is plan 014's bug, symptom first: an
|
||||
// explore_index created before `total_tracks` existed, met by the
|
||||
// projection every explore read uses. Before the repair this failed
|
||||
// with "no such column: total_tracks" on every install that already had
|
||||
// a catalog, while a fresh one was perfectly healthy.
|
||||
func TestRetiresIndexMissingAColumn(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
dir := t.TempDir()
|
||||
db := openRaw(t, dir)
|
||||
|
||||
// The pre-014 shape: the columns the projection needs, minus the
|
||||
// one the plan added.
|
||||
if _, err := db.ExecContext(ctx, `
|
||||
CREATE TABLE explore_index (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
entity_type INTEGER NOT NULL,
|
||||
mbid BLOB NOT NULL,
|
||||
title TEXT NOT NULL,
|
||||
artist_name TEXT NOT NULL,
|
||||
artist_mbid BLOB NOT NULL
|
||||
);
|
||||
INSERT INTO explore_index (entity_type, mbid, title, artist_name, artist_mbid)
|
||||
VALUES (1, x'00112233445566778899aabbccddeeff', 'x', 'y', x'');
|
||||
`); err != nil {
|
||||
t.Fatalf("seed: %v", err)
|
||||
}
|
||||
|
||||
if err := retireStaleTables(ctx, db, testLogger()); err != nil {
|
||||
t.Fatalf("retire: %v", err)
|
||||
}
|
||||
|
||||
if err := applySchema(ctx, db); err != nil {
|
||||
t.Fatalf("applySchema: %v", err)
|
||||
}
|
||||
|
||||
// The column the projection needs is there now.
|
||||
var n int
|
||||
if err := db.QueryRowContext(ctx,
|
||||
`SELECT COUNT(*) FROM pragma_table_info('explore_index')
|
||||
WHERE name = 'total_tracks'`,
|
||||
).Scan(&n); err != nil {
|
||||
t.Fatalf("inspect: %v", err)
|
||||
}
|
||||
|
||||
if n != 1 {
|
||||
t.Fatalf("explore_index still has no total_tracks column")
|
||||
}
|
||||
|
||||
// And the catalog really was retired rather than patched, so the
|
||||
// artifact is fetched again instead of half a catalog being served.
|
||||
if err := db.QueryRowContext(ctx,
|
||||
"SELECT COUNT(*) FROM explore_index",
|
||||
).Scan(&n); err != nil {
|
||||
t.Fatalf("count: %v", err)
|
||||
}
|
||||
|
||||
if n != 0 {
|
||||
t.Fatalf("stale rows survived the retire: %d", n)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRetiresIndexWithTextMBIDs is the half an ALTER could not have
|
||||
// repaired: plan 013 changed mbid from TEXT to BLOB, and SQLite does not
|
||||
// coerce between them, so a query against 16 raw bytes returns no rows
|
||||
// rather than an error.
|
||||
func TestRetiresIndexWithTextMBIDs(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
dir := t.TempDir()
|
||||
db := openRaw(t, dir)
|
||||
|
||||
if _, err := db.ExecContext(ctx, `
|
||||
CREATE TABLE explore_index (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
entity_type TEXT NOT NULL,
|
||||
mbid TEXT NOT NULL,
|
||||
title TEXT NOT NULL,
|
||||
artist_name TEXT NOT NULL,
|
||||
artist_mbid TEXT NOT NULL,
|
||||
total_tracks INTEGER NOT NULL DEFAULT 0
|
||||
);
|
||||
`); err != nil {
|
||||
t.Fatalf("seed: %v", err)
|
||||
}
|
||||
|
||||
if err := retireStaleTables(ctx, db, testLogger()); err != nil {
|
||||
t.Fatalf("retire: %v", err)
|
||||
}
|
||||
|
||||
if err := applySchema(ctx, db); err != nil {
|
||||
t.Fatalf("applySchema: %v", err)
|
||||
}
|
||||
|
||||
var typ string
|
||||
if err := db.QueryRowContext(ctx,
|
||||
`SELECT type FROM pragma_table_info('explore_index') WHERE name = 'mbid'`,
|
||||
).Scan(&typ); err != nil {
|
||||
t.Fatalf("inspect: %v", err)
|
||||
}
|
||||
|
||||
if typ != "BLOB" {
|
||||
t.Fatalf("mbid is still %s, want BLOB", typ)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRetiringTheIndexTakesItsMetaWithIt guards the thing that makes the
|
||||
// repair actually repair: the marker saying the import finished is what
|
||||
// stops the artifact being fetched again, so a catalog dropped without
|
||||
// it would stay empty forever.
|
||||
func TestRetiringTheIndexTakesItsMetaWithIt(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
dir := t.TempDir()
|
||||
db := openRaw(t, dir)
|
||||
|
||||
if _, err := db.ExecContext(ctx, `
|
||||
CREATE TABLE explore_index (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
entity_type INTEGER NOT NULL,
|
||||
mbid BLOB NOT NULL
|
||||
);
|
||||
CREATE TABLE explore_index_meta (key TEXT PRIMARY KEY, value TEXT NOT NULL);
|
||||
INSERT INTO explore_index_meta VALUES ('dump_import_done', '1');
|
||||
`); err != nil {
|
||||
t.Fatalf("seed: %v", err)
|
||||
}
|
||||
|
||||
if err := retireStaleTables(ctx, db, testLogger()); err != nil {
|
||||
t.Fatalf("retire: %v", err)
|
||||
}
|
||||
|
||||
if err := applySchema(ctx, db); err != nil {
|
||||
t.Fatalf("applySchema: %v", err)
|
||||
}
|
||||
|
||||
var n int
|
||||
if err := db.QueryRowContext(ctx,
|
||||
"SELECT COUNT(*) FROM explore_index_meta WHERE key = 'dump_import_done'",
|
||||
).Scan(&n); err != nil {
|
||||
t.Fatalf("meta: %v", err)
|
||||
}
|
||||
|
||||
if n != 0 {
|
||||
t.Fatalf("the import-done marker survived a retired catalog")
|
||||
}
|
||||
}
|
||||
|
||||
// TestHealthyDatabaseIsUntouched is the other half, and the one that
|
||||
// would make this dangerous if it failed: a current schema must survive
|
||||
// a launch with its catalog intact. A repair that retires a healthy
|
||||
// catalog costs every user an artifact download on every start.
|
||||
func TestHealthyDatabaseIsUntouched(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
dir := t.TempDir()
|
||||
db := openRaw(t, dir)
|
||||
|
||||
if err := applySchema(ctx, db); err != nil {
|
||||
t.Fatalf("applySchema: %v", err)
|
||||
}
|
||||
|
||||
if _, err := db.ExecContext(ctx, `
|
||||
INSERT INTO explore_index (entity_type, mbid, title, artist_name, artist_mbid)
|
||||
VALUES (1, x'00112233445566778899aabbccddeeff', 'x', 'y', x'')
|
||||
`); err != nil {
|
||||
t.Fatalf("seed: %v", err)
|
||||
}
|
||||
|
||||
if err := retireStaleTables(ctx, db, testLogger()); err != nil {
|
||||
t.Fatalf("retire: %v", err)
|
||||
}
|
||||
|
||||
var n int
|
||||
if err := db.QueryRowContext(ctx,
|
||||
"SELECT COUNT(*) FROM explore_index",
|
||||
).Scan(&n); err != nil {
|
||||
t.Fatalf("count: %v", err)
|
||||
}
|
||||
|
||||
if n != 1 {
|
||||
t.Fatalf("a healthy catalog was retired: %d rows left", n)
|
||||
}
|
||||
}
|
||||
|
||||
// TestAuthoredTablesAreNeverRetired states the boundary in a test rather
|
||||
// than only in a comment: this mechanism deletes data, and the only
|
||||
// thing standing between it and a user's playlists is the Kind filter.
|
||||
func TestAuthoredTablesAreNeverRetired(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
dir := t.TempDir()
|
||||
db := openRaw(t, dir)
|
||||
|
||||
// A playlists table missing most of its current columns.
|
||||
if _, err := db.ExecContext(ctx, `
|
||||
CREATE TABLE playlists (id INTEGER PRIMARY KEY, name TEXT NOT NULL);
|
||||
INSERT INTO playlists (name) VALUES ('irreplaceable');
|
||||
`); err != nil {
|
||||
t.Fatalf("seed: %v", err)
|
||||
}
|
||||
|
||||
if err := retireStaleTables(ctx, db, testLogger()); err != nil {
|
||||
t.Fatalf("retire: %v", err)
|
||||
}
|
||||
|
||||
var n int
|
||||
if err := db.QueryRowContext(ctx,
|
||||
"SELECT COUNT(*) FROM playlists",
|
||||
).Scan(&n); err != nil {
|
||||
t.Fatalf("count: %v", err)
|
||||
}
|
||||
|
||||
if n != 1 {
|
||||
t.Fatalf("an authored table was retired; rows left: %d", n)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRetiresTablesTheSchemaNoLongerDescribes covers what plan 013 left
|
||||
// behind on every database that predates it: seven tables the schema
|
||||
// stopped describing, plus the schema_migrations table that squashing
|
||||
// the chain retired. They are not stale in shape — they are simply not
|
||||
// ours any more.
|
||||
func TestRetiresTablesTheSchemaNoLongerDescribes(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
db := openRaw(t, t.TempDir())
|
||||
|
||||
if _, err := db.ExecContext(ctx, `
|
||||
CREATE TABLE recordings (id INTEGER PRIMARY KEY, name TEXT);
|
||||
CREATE TABLE artist_credit (id INTEGER PRIMARY KEY, text TEXT);
|
||||
CREATE TABLE schema_migrations (version INTEGER PRIMARY KEY);
|
||||
`); err != nil {
|
||||
t.Fatalf("seed: %v", err)
|
||||
}
|
||||
|
||||
if err := retireStaleTables(ctx, db, testLogger()); err != nil {
|
||||
t.Fatalf("retire: %v", err)
|
||||
}
|
||||
|
||||
for _, table := range []string{"recordings", "artist_credit", "schema_migrations"} {
|
||||
var n int
|
||||
|
||||
if err := db.QueryRowContext(ctx,
|
||||
"SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name = ?",
|
||||
table,
|
||||
).Scan(&n); err != nil {
|
||||
t.Fatalf("inspect %s: %v", table, err)
|
||||
}
|
||||
|
||||
if n != 0 {
|
||||
t.Errorf("%s survived; the schema no longer describes it", table)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestFTSShadowTablesAreNotObsolete is the sweep's sharp edge: an FTS5
|
||||
// virtual table is backed by four shadow tables that appear in
|
||||
// sqlite_master under their own names and are in no schema file.
|
||||
// Dropping one destroys the index it belongs to.
|
||||
func TestFTSShadowTablesAreNotObsolete(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
db := openRaw(t, t.TempDir())
|
||||
|
||||
if err := applySchema(ctx, db); err != nil {
|
||||
t.Fatalf("applySchema: %v", err)
|
||||
}
|
||||
|
||||
obsolete, err := obsoleteTables(ctx, db)
|
||||
if err != nil {
|
||||
t.Fatalf("obsoleteTables: %v", err)
|
||||
}
|
||||
|
||||
if len(obsolete) != 0 {
|
||||
t.Fatalf("a freshly created schema reported obsolete tables: %v", obsolete)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRetiringOwnedTablesDoesNotDangle pins the one behaviour that is
|
||||
// silently wrong rather than loudly broken.
|
||||
//
|
||||
// Retiring audio_files leaves playlist entries behind. With
|
||||
// foreign_keys ON — which applyPRAGMAs has done before NewDB gets here —
|
||||
// SET NULL fires and they point at nothing. With it OFF they keep ids
|
||||
// that the rescan reissues from 1, so every playlist quietly fills with
|
||||
// different songs. Nothing about the schema makes that ordering
|
||||
// obvious, so it is asserted rather than assumed.
|
||||
func TestRetiringOwnedTablesDoesNotDangle(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
db := openRaw(t, t.TempDir())
|
||||
|
||||
if _, err := db.ExecContext(ctx, "PRAGMA foreign_keys = ON"); err != nil {
|
||||
t.Fatalf("pragma: %v", err)
|
||||
}
|
||||
|
||||
if err := applySchema(ctx, db); err != nil {
|
||||
t.Fatalf("applySchema: %v", err)
|
||||
}
|
||||
|
||||
// Break audio_files' shape so it is retired, keeping a playlist
|
||||
// entry that references it.
|
||||
if _, err := db.ExecContext(ctx, `
|
||||
INSERT INTO playlists (id, name) VALUES (1, 'keepme');
|
||||
INSERT INTO libraries (id, name, path) VALUES (0, 'test', '/music');
|
||||
INSERT INTO audio_files (id, file_path, file_type_id, length_milliseconds)
|
||||
VALUES (7, '/music/a.flac', 1, 1000);
|
||||
INSERT INTO playlist_tracks (playlist_id, audio_file_id, position)
|
||||
VALUES (1, 7, 0);
|
||||
DROP VIEW IF EXISTS track_metadata;
|
||||
ALTER TABLE audio_files DROP COLUMN artist_credit;
|
||||
`); err != nil {
|
||||
t.Fatalf("seed: %v", err)
|
||||
}
|
||||
|
||||
if err := retireStaleTables(ctx, db, testLogger()); err != nil {
|
||||
t.Fatalf("retire: %v", err)
|
||||
}
|
||||
|
||||
if err := applySchema(ctx, db); err != nil {
|
||||
t.Fatalf("applySchema: %v", err)
|
||||
}
|
||||
|
||||
var dangling int
|
||||
if err := db.QueryRowContext(ctx,
|
||||
"SELECT COUNT(*) FROM playlist_tracks WHERE audio_file_id IS NOT NULL",
|
||||
).Scan(&dangling); err != nil {
|
||||
t.Fatalf("count: %v", err)
|
||||
}
|
||||
|
||||
if dangling != 0 {
|
||||
t.Fatalf(
|
||||
"%d playlist entries still point at retired audio_files ids; "+
|
||||
"a rescan will reissue those ids to different tracks",
|
||||
dangling,
|
||||
)
|
||||
}
|
||||
|
||||
// The playlist itself is authored and must be untouched.
|
||||
var playlists int
|
||||
if err := db.QueryRowContext(ctx,
|
||||
"SELECT COUNT(*) FROM playlists",
|
||||
).Scan(&playlists); err != nil {
|
||||
t.Fatalf("playlists: %v", err)
|
||||
}
|
||||
|
||||
if playlists != 1 {
|
||||
t.Fatalf("authored playlist lost: %d", playlists)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRetiringInterlinkedLegacyTables is the bug the unit tests missed
|
||||
// and a real database found.
|
||||
//
|
||||
// The tables plan 013 retired reference each other -- pre-013
|
||||
// audio_files has a foreign key into recordings -- so with foreign keys
|
||||
// ON, dropping them one at a time fails with "FOREIGN KEY constraint
|
||||
// failed" on whichever goes first, and map iteration order decides
|
||||
// which that is. Every other test in this file ran with foreign keys
|
||||
// off and passed happily; the app enables them in applyPRAGMAs before
|
||||
// the repair runs, so only the real launch path showed it.
|
||||
func TestRetiringInterlinkedLegacyTables(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
db := openRaw(t, t.TempDir())
|
||||
|
||||
if _, err := db.ExecContext(ctx, "PRAGMA foreign_keys = ON"); err != nil {
|
||||
t.Fatalf("pragma: %v", err)
|
||||
}
|
||||
|
||||
// The pre-013 shape, with the reference that makes ordering matter.
|
||||
// release_group_recordings sorts *after* recordings and references
|
||||
// it, so the deterministic order retires the parent while the child
|
||||
// still holds rows pointing at it -- which is the case that fails
|
||||
// without the deferral, rather than one that fails on some runs.
|
||||
if _, err := db.ExecContext(ctx, `
|
||||
CREATE TABLE recordings (id INTEGER PRIMARY KEY, name TEXT);
|
||||
CREATE TABLE artist_credit (id INTEGER PRIMARY KEY, text TEXT);
|
||||
CREATE TABLE release_group_recordings (
|
||||
id INTEGER PRIMARY KEY,
|
||||
recording_id INTEGER NOT NULL,
|
||||
FOREIGN KEY(recording_id) REFERENCES recordings(id)
|
||||
);
|
||||
CREATE TABLE audio_files (
|
||||
id INTEGER PRIMARY KEY,
|
||||
file_path TEXT NOT NULL UNIQUE,
|
||||
recording_id INTEGER,
|
||||
FOREIGN KEY(recording_id) REFERENCES recordings(id)
|
||||
);
|
||||
INSERT INTO recordings (id, name) VALUES (1, 'x');
|
||||
INSERT INTO release_group_recordings (id, recording_id) VALUES (1, 1);
|
||||
INSERT INTO audio_files (id, file_path, recording_id)
|
||||
VALUES (1, '/music/a.flac', 1);
|
||||
`); err != nil {
|
||||
t.Fatalf("seed: %v", err)
|
||||
}
|
||||
|
||||
if err := retireStaleTables(ctx, db, testLogger()); err != nil {
|
||||
t.Fatalf("retire: %v", err)
|
||||
}
|
||||
|
||||
if err := applySchema(ctx, db); err != nil {
|
||||
t.Fatalf("applySchema: %v", err)
|
||||
}
|
||||
|
||||
for _, table := range []string{"recordings", "artist_credit"} {
|
||||
var n int
|
||||
|
||||
if err := db.QueryRowContext(ctx,
|
||||
"SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name = ?",
|
||||
table,
|
||||
).Scan(&n); err != nil {
|
||||
t.Fatalf("inspect %s: %v", table, err)
|
||||
}
|
||||
|
||||
if n != 0 {
|
||||
t.Errorf("%s survived the retire", table)
|
||||
}
|
||||
}
|
||||
|
||||
// And the rebuilt audio_files is the current shape, which is the
|
||||
// whole reason the old one had to go.
|
||||
var n int
|
||||
if err := db.QueryRowContext(ctx,
|
||||
`SELECT COUNT(*) FROM pragma_table_info('audio_files')
|
||||
WHERE name = 'artist_credit'`,
|
||||
).Scan(&n); err != nil {
|
||||
t.Fatalf("inspect audio_files: %v", err)
|
||||
}
|
||||
|
||||
if n != 1 {
|
||||
t.Fatal("audio_files was not rebuilt in the current shape")
|
||||
}
|
||||
}
|
||||
|
||||
// TestParseCreateTablesReadsTheRealSchema keeps the parser honest
|
||||
// against the files it actually runs on: a parser that silently found
|
||||
// no columns would report every table healthy and repair nothing.
|
||||
func TestParseCreateTablesReadsTheRealSchema(t *testing.T) {
|
||||
declared, err := declaredTables()
|
||||
if err != nil {
|
||||
t.Fatalf("declaredTables: %v", err)
|
||||
}
|
||||
|
||||
cols, ok := declared["explore_index"]
|
||||
if !ok {
|
||||
t.Fatal("explore_index was not parsed out of the schema files")
|
||||
}
|
||||
|
||||
want := map[string]string{
|
||||
"mbid": "BLOB",
|
||||
"total_tracks": "INTEGER",
|
||||
"artist_name": "TEXT",
|
||||
}
|
||||
|
||||
got := make(map[string]string, len(cols))
|
||||
for _, c := range cols {
|
||||
got[c.name] = c.typ
|
||||
}
|
||||
|
||||
for name, typ := range want {
|
||||
if got[name] != typ {
|
||||
t.Errorf("explore_index.%s parsed as %q, want %q", name, got[name], typ)
|
||||
}
|
||||
}
|
||||
|
||||
// A table constraint must not be mistaken for a column.
|
||||
for _, c := range cols {
|
||||
switch c.name {
|
||||
case "PRIMARY", "FOREIGN", "UNIQUE", "CHECK", "CONSTRAINT":
|
||||
t.Errorf("parsed table constraint %q as a column", c.name)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -185,6 +185,21 @@ var tables = []Table{
|
||||
Note: "Full-text index over the champion entities of the " +
|
||||
"MusicBrainz dump. Rebuilt only by a full index build.",
|
||||
},
|
||||
{
|
||||
Name: "artist_credit_part", Kind: Cache, Lifetime: Retained,
|
||||
Note: "The decomposition of a multi-artist credit, from the " +
|
||||
"MusicBrainz dump: one row per credited artist, with the " +
|
||||
"name as credited and the join phrase that follows it. " +
|
||||
"Arrives with the downloaded artifact, so rebuilding it " +
|
||||
"costs a dump stream and it is never swept.",
|
||||
},
|
||||
{
|
||||
Name: "artist_credit_ref", Kind: Cache, Lifetime: Retained,
|
||||
Note: "Which credit a catalog recording or release group is " +
|
||||
"credited to. Present only for multi-artist credits; " +
|
||||
"absence means one artist, which explore_index already " +
|
||||
"describes. Ships and dies with artist_credit_part.",
|
||||
},
|
||||
{
|
||||
Name: "explore_index", Kind: Cache, Lifetime: Retained,
|
||||
Note: "The offline MusicBrainz search index. Rebuilding costs a " +
|
||||
|
||||
@@ -199,24 +199,23 @@ func TestManagerEndToEndAutoPick(t *testing.T) {
|
||||
t.Errorf("expected imported file at %s: %v", want, err)
|
||||
}
|
||||
|
||||
// Staging was released only after a successful import.
|
||||
entries, err := os.ReadDir(f.staging.Root())
|
||||
if err != nil {
|
||||
t.Fatalf("read staging root: %v", err)
|
||||
}
|
||||
// Staging release and the rescan happen *after* the state is
|
||||
// recorded (manager.go sets StateComplete, then releases, then
|
||||
// scans), so waiting on the state is not waiting on these. Under
|
||||
// load the worker is descheduled in between and asserting straight
|
||||
// away reads the world one step too early -- which is exactly how
|
||||
// this test failed on a busy machine while passing alone.
|
||||
waitFor(t, func() bool {
|
||||
entries, err := os.ReadDir(f.staging.Root())
|
||||
if err != nil || len(entries) != 0 {
|
||||
return false
|
||||
}
|
||||
|
||||
if len(entries) != 0 {
|
||||
t.Errorf("staging not released: %d dirs remain", len(entries))
|
||||
}
|
||||
f.lib.mu.Lock()
|
||||
defer f.lib.mu.Unlock()
|
||||
|
||||
// The library was told to rescan.
|
||||
f.lib.mu.Lock()
|
||||
scanned := len(f.lib.scanned)
|
||||
f.lib.mu.Unlock()
|
||||
|
||||
if scanned != 1 {
|
||||
t.Errorf("library scans = %d, want 1", scanned)
|
||||
}
|
||||
return len(f.lib.scanned) == 1
|
||||
}, "staging was never released, or the library was never rescanned")
|
||||
}
|
||||
|
||||
// An ambiguous result set must park for the user rather than guess.
|
||||
|
||||
@@ -283,6 +283,9 @@ func (si *SearchIndex) importCoreArtifact(ctx context.Context, path string) erro
|
||||
}
|
||||
|
||||
merged, mergeErr := si.mergeArtifactRows(ctx, info.rows)
|
||||
if mergeErr == nil {
|
||||
si.mergeArtifactCredits(ctx)
|
||||
}
|
||||
|
||||
if ftsSuspended {
|
||||
start := time.Now()
|
||||
@@ -473,3 +476,75 @@ func (si *SearchIndex) removeArtifactFile(path string) {
|
||||
si.logger.Warn("core artifact: cleanup failed", "path", path, "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
// artifactHasCredits reports whether the attached artifact carries the
|
||||
// multi-artist credit tables.
|
||||
//
|
||||
// The same shape, and the same handle, as artifactHasTotals above: an
|
||||
// artifact published before credits existed is still a perfectly good
|
||||
// catalog, and there is one already out there. Selecting from a table
|
||||
// that is not in it would fail an import that should have succeeded, so
|
||||
// it is asked rather than assumed -- on the *writer*, because `core` is
|
||||
// attached to that one connection and the read pool cannot see it.
|
||||
func (si *SearchIndex) artifactHasCredits() bool {
|
||||
var n int
|
||||
|
||||
err := si.db.QueryRowWriter(
|
||||
`SELECT COUNT(*) FROM core.sqlite_master
|
||||
WHERE type = 'table' AND name IN ('artist_credit_part', 'artist_credit_ref')`,
|
||||
).Scan(&n)
|
||||
|
||||
return err == nil && n == 2
|
||||
}
|
||||
|
||||
// mergeArtifactCredits copies the credit decomposition out of the
|
||||
// attached artifact.
|
||||
//
|
||||
// Credits are replaced wholesale rather than merged: they are derived
|
||||
// entirely from one dump build, they are keyed by ids that are only
|
||||
// meaningful within the artifact that carried them, and a half-updated
|
||||
// credit renders as the wrong artists rather than as missing ones.
|
||||
//
|
||||
// A failure here is logged and not returned. The catalog has already
|
||||
// merged at this point, and a catalog without credits is the catalog
|
||||
// this app had before them -- every credit falls back to its single
|
||||
// artist, which is the same fallback an untagged file already gets.
|
||||
func (si *SearchIndex) mergeArtifactCredits(ctx context.Context) {
|
||||
if !si.artifactHasCredits() {
|
||||
si.logger.Info("core artifact: no credit tables, keeping single-artist credits")
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
start := time.Now()
|
||||
|
||||
for _, stmt := range []string{
|
||||
"DELETE FROM artist_credit_part",
|
||||
"DELETE FROM artist_credit_ref",
|
||||
`INSERT OR REPLACE INTO artist_credit_part
|
||||
(credit_id, position, artist_mbid, credited_name, join_phrase)
|
||||
SELECT credit_id, position, artist_mbid, credited_name, join_phrase
|
||||
FROM core.artist_credit_part`,
|
||||
`INSERT OR REPLACE INTO artist_credit_ref (mbid, credit_id)
|
||||
SELECT mbid, credit_id FROM core.artist_credit_ref`,
|
||||
} {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return
|
||||
}
|
||||
|
||||
if _, err := si.db.ExecContext(stmt); err != nil {
|
||||
si.logger.Warn("core artifact: credit merge failed", "error", err)
|
||||
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
var refs int
|
||||
|
||||
_ = si.db.QueryRowWriter("SELECT COUNT(*) FROM artist_credit_ref").Scan(&refs)
|
||||
|
||||
si.logger.Info("core artifact: credits merged",
|
||||
"entities", refs,
|
||||
"elapsed", time.Since(start).Round(time.Millisecond),
|
||||
)
|
||||
}
|
||||
|
||||
@@ -3,6 +3,7 @@ package explore
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/hex"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
@@ -597,3 +598,153 @@ func TestImportCoreArtifactReadsTotalsWhenPresent(t *testing.T) {
|
||||
t.Errorf("TotalTracks = %d, want 0 (the catalog does not say)", old.TotalTracks)
|
||||
}
|
||||
}
|
||||
|
||||
// addArtifactCredits gives an artifact file the credit tables the
|
||||
// exporter now writes, so the import path can be exercised against one
|
||||
// that has them.
|
||||
func addArtifactCredits(t *testing.T, path string) {
|
||||
t.Helper()
|
||||
|
||||
db, err := sql.Open("sqlite", "file:"+path)
|
||||
if err != nil {
|
||||
t.Fatalf("open artifact: %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = db.Close() }()
|
||||
|
||||
for _, stmt := range []string{
|
||||
`CREATE TABLE artist_credit_part (
|
||||
credit_id INTEGER NOT NULL,
|
||||
position INTEGER NOT NULL,
|
||||
artist_mbid BLOB NOT NULL,
|
||||
credited_name TEXT NOT NULL,
|
||||
join_phrase TEXT NOT NULL DEFAULT '',
|
||||
PRIMARY KEY (credit_id, position)
|
||||
) WITHOUT ROWID`,
|
||||
`CREATE TABLE artist_credit_ref (
|
||||
mbid BLOB NOT NULL PRIMARY KEY,
|
||||
credit_id INTEGER NOT NULL
|
||||
) WITHOUT ROWID`,
|
||||
} {
|
||||
if _, err := db.Exec(stmt); err != nil {
|
||||
t.Fatalf("create credit tables: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// The packed form the catalog stores. uuid16/parseUUID live behind
|
||||
// the indexbuild tag, so this file decodes for itself.
|
||||
pack := func(mbid string) []byte {
|
||||
raw, err := hex.DecodeString(strings.ReplaceAll(mbid, "-", ""))
|
||||
if err != nil || len(raw) != 16 {
|
||||
t.Fatalf("fixture MBID %q is not a UUID: %v", mbid, err)
|
||||
}
|
||||
|
||||
return raw
|
||||
}
|
||||
|
||||
a, b, rec := pack(artA), pack(artB), pack(recA)
|
||||
|
||||
for _, part := range [][]any{
|
||||
{7, 0, a, "Artist A", " feat. "},
|
||||
{7, 1, b, "Artist B", ""},
|
||||
} {
|
||||
if _, err := db.Exec(`INSERT INTO artist_credit_part
|
||||
(credit_id, position, artist_mbid, credited_name, join_phrase)
|
||||
VALUES (?, ?, ?, ?, ?)`, part...); err != nil {
|
||||
t.Fatalf("insert part: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
if _, err := db.Exec(
|
||||
"INSERT INTO artist_credit_ref (mbid, credit_id) VALUES (?, ?)", rec, 7,
|
||||
); err != nil {
|
||||
t.Fatalf("insert ref: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestImportCoreArtifactMergesCredits is the positive half of the
|
||||
// compatibility pair: an artifact that carries credits delivers them,
|
||||
// rendering back to the credit string they decompose.
|
||||
func TestImportCoreArtifactMergesCredits(t *testing.T) {
|
||||
db := database.NewTestDB(t)
|
||||
si := NewSearchIndex(db, nil, nil, testLogger())
|
||||
|
||||
path := writeTestArtifact(t, validMeta(), []artifactRow{
|
||||
{"recording", recA, "Song A", "Artist A feat. Artist B", artA, 2000},
|
||||
})
|
||||
|
||||
addArtifactCredits(t, path)
|
||||
|
||||
if err := si.importCoreArtifact(context.Background(), path); err != nil {
|
||||
t.Fatalf("importCoreArtifact: %v", err)
|
||||
}
|
||||
|
||||
rows, err := db.QueryContext(
|
||||
`SELECT p.credited_name, p.join_phrase
|
||||
FROM artist_credit_ref r
|
||||
JOIN artist_credit_part p ON p.credit_id = r.credit_id
|
||||
ORDER BY p.position`,
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("query credits: %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
var rendered strings.Builder
|
||||
|
||||
for rows.Next() {
|
||||
var name, join string
|
||||
|
||||
if err := rows.Scan(&name, &join); err != nil {
|
||||
t.Fatalf("scan: %v", err)
|
||||
}
|
||||
|
||||
rendered.WriteString(name)
|
||||
rendered.WriteString(join)
|
||||
}
|
||||
|
||||
if got := rendered.String(); got != "Artist A feat. Artist B" {
|
||||
t.Errorf("rendered credit = %q, want %q", got, "Artist A feat. Artist B")
|
||||
}
|
||||
}
|
||||
|
||||
// TestImportCoreArtifactWithoutCredits is the regression that matters
|
||||
// most here: an artifact published before credits existed cannot be
|
||||
// re-cut retroactively, so it must import as a catalog that declines to
|
||||
// answer rather than failing outright. writeTestArtifact deliberately
|
||||
// builds one without the tables.
|
||||
func TestImportCoreArtifactWithoutCredits(t *testing.T) {
|
||||
db := database.NewTestDB(t)
|
||||
si := NewSearchIndex(db, nil, nil, testLogger())
|
||||
|
||||
path := writeTestArtifact(t, validMeta(), []artifactRow{
|
||||
{"recording", recA, "Song A", "Artist A", artA, 2000},
|
||||
})
|
||||
|
||||
if err := si.importCoreArtifact(context.Background(), path); err != nil {
|
||||
t.Fatalf("an artifact without credit tables must still import: %v", err)
|
||||
}
|
||||
|
||||
var rows int
|
||||
if err := db.QueryRowWriter(
|
||||
"SELECT COUNT(*) FROM explore_index",
|
||||
).Scan(&rows); err != nil {
|
||||
t.Fatalf("count: %v", err)
|
||||
}
|
||||
|
||||
if rows != 1 {
|
||||
t.Errorf("catalog rows = %d, want 1", rows)
|
||||
}
|
||||
|
||||
var refs int
|
||||
if err := db.QueryRowWriter(
|
||||
"SELECT COUNT(*) FROM artist_credit_ref",
|
||||
).Scan(&refs); err != nil {
|
||||
t.Fatalf("count refs: %v", err)
|
||||
}
|
||||
|
||||
if refs != 0 {
|
||||
t.Errorf("credit refs = %d, want 0", refs)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,132 @@
|
||||
package explore
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Reading multi-artist credits back out of the catalog.
|
||||
//
|
||||
// The tables are filled centrally (backend/explore/dumpcredits.go, and
|
||||
// the artifact import) and hold only credits naming more than one
|
||||
// artist: an entity with no rows here is credited to one artist, which
|
||||
// explore_index's own artist_name and artist_mbid already describe.
|
||||
// Absence is the common case and means "nothing to decompose", never
|
||||
// "unknown".
|
||||
//
|
||||
// The lookup is keyed on the *recording* MBID, which both sides of the
|
||||
// app already have -- a catalog row carries it and so does a local
|
||||
// file (library.Track.RecordingMBID) -- so one query serves the Explore
|
||||
// pages and the library's own lists without either needing to know
|
||||
// where the other gets its rows.
|
||||
|
||||
// CreditPart is one credited artist within a credit, in credit order.
|
||||
//
|
||||
// CreditedName is the name *as credited*, which is not the artist's own
|
||||
// name: MusicBrainz credits "Snoop Dogg" on a track by the artist
|
||||
// called "Snoop Doggy Dogg". Display uses it; navigation uses
|
||||
// ArtistMBID. JoinPhrase is the literal connector that follows this
|
||||
// part, so a credit renders by concatenation and never by searching a
|
||||
// name inside a credit string.
|
||||
type CreditPart struct {
|
||||
Position int `json:"position"`
|
||||
ArtistMBID string `json:"artistMbid"`
|
||||
CreditedName string `json:"creditedName"`
|
||||
JoinPhrase string `json:"joinPhrase"`
|
||||
}
|
||||
|
||||
// creditLookupBatch bounds how many MBIDs go into one IN clause. A
|
||||
// tracklist is the caller here, so the realistic ceiling is a few
|
||||
// hundred; the bound exists so a 50,000-row selection cannot build a
|
||||
// statement SQLite refuses to parse.
|
||||
const creditLookupBatch = 500
|
||||
|
||||
// GetCredits returns the decomposition of every multi-artist credit
|
||||
// among the given entity MBIDs, keyed by MBID.
|
||||
//
|
||||
// MBIDs with a single-artist credit are simply absent from the result,
|
||||
// which is what the caller wants: it renders its existing single link
|
||||
// for those, and that is the same answer it would have rendered anyway.
|
||||
func (si *SearchIndex) GetCredits(mbids []string) (map[string][]CreditPart, error) {
|
||||
out := make(map[string][]CreditPart)
|
||||
|
||||
for start := 0; start < len(mbids); start += creditLookupBatch {
|
||||
end := min(start+creditLookupBatch, len(mbids))
|
||||
|
||||
if err := si.appendCredits(mbids[start:end], out); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// appendCredits runs one batch into the accumulating result.
|
||||
func (si *SearchIndex) appendCredits(
|
||||
mbids []string, out map[string][]CreditPart,
|
||||
) error {
|
||||
args := make([]any, 0, len(mbids))
|
||||
holders := make([]string, 0, len(mbids))
|
||||
|
||||
for _, mbid := range mbids {
|
||||
if mbid == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
args = append(args, dbMBID(mbid))
|
||||
holders = append(holders, "?")
|
||||
}
|
||||
|
||||
if len(args) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Ordered by position because that ordering *is* the credit's
|
||||
// meaning; the caller concatenates in the order it receives.
|
||||
rows, err := si.db.QueryContext(
|
||||
`SELECT r.mbid, p.position, p.artist_mbid, p.credited_name, p.join_phrase
|
||||
FROM artist_credit_ref r
|
||||
JOIN artist_credit_part p ON p.credit_id = r.credit_id
|
||||
WHERE r.mbid IN (`+strings.Join(holders, ",")+`)
|
||||
ORDER BY r.mbid, p.position`,
|
||||
args...,
|
||||
)
|
||||
if err != nil {
|
||||
return fmt.Errorf("read artist credits: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
for rows.Next() {
|
||||
var (
|
||||
entity dbMBID
|
||||
artist dbMBID
|
||||
part CreditPart
|
||||
)
|
||||
|
||||
if err := rows.Scan(
|
||||
&entity, &part.Position, &artist, &part.CreditedName, &part.JoinPhrase,
|
||||
); err != nil {
|
||||
return fmt.Errorf("scan artist credit: %w", err)
|
||||
}
|
||||
|
||||
part.ArtistMBID = string(artist)
|
||||
out[string(entity)] = append(out[string(entity)], part)
|
||||
}
|
||||
|
||||
if err := rows.Err(); err != nil {
|
||||
return fmt.Errorf("read artist credits: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetCredits is the bound form: the frontend asks for a tracklist's
|
||||
// worth of MBIDs at once rather than one per row.
|
||||
//
|
||||
// Batched for the reason every other per-row backend question here is:
|
||||
// asking on hover or on render turns a list into N IPC round trips, and
|
||||
// this one is asked about every row of every list in the app.
|
||||
func (e *Service) GetCredits(mbids []string) (map[string][]CreditPart, error) {
|
||||
return e.index.GetCredits(mbids)
|
||||
}
|
||||
@@ -0,0 +1,128 @@
|
||||
package explore
|
||||
|
||||
import (
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"yellowjacket/backend/database"
|
||||
)
|
||||
|
||||
// seedCredit writes one multi-artist credit and points an entity at it,
|
||||
// the way the dump import and the artifact import both do.
|
||||
func seedCredit(t *testing.T, db *database.DB, entity string, id int, parts []CreditPart) {
|
||||
t.Helper()
|
||||
|
||||
pack := func(mbid string) []byte {
|
||||
raw, err := hex.DecodeString(strings.ReplaceAll(mbid, "-", ""))
|
||||
if err != nil || len(raw) != 16 {
|
||||
t.Fatalf("bad fixture mbid %q: %v", mbid, err)
|
||||
}
|
||||
|
||||
return raw
|
||||
}
|
||||
|
||||
if _, err := db.ExecContext(
|
||||
"INSERT INTO artist_credit_ref (mbid, credit_id) VALUES (?, ?)",
|
||||
pack(entity), id,
|
||||
); err != nil {
|
||||
t.Fatalf("seed ref: %v", err)
|
||||
}
|
||||
|
||||
for _, p := range parts {
|
||||
if _, err := db.ExecContext(
|
||||
`INSERT INTO artist_credit_part
|
||||
(credit_id, position, artist_mbid, credited_name, join_phrase)
|
||||
VALUES (?, ?, ?, ?, ?)`,
|
||||
id, p.Position, pack(p.ArtistMBID), p.CreditedName, p.JoinPhrase,
|
||||
); err != nil {
|
||||
t.Fatalf("seed part: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestGetCreditsDecomposes: the parts come back in position order and
|
||||
// concatenate to the credit they describe.
|
||||
func TestGetCreditsDecomposes(t *testing.T) {
|
||||
db := database.NewTestDB(t)
|
||||
si := NewSearchIndex(db, nil, nil, testLogger())
|
||||
|
||||
rec := testMBID("rec-1")
|
||||
a, b := testMBID("artist-a"), testMBID("artist-b")
|
||||
|
||||
seedCredit(t, db, rec, 7, []CreditPart{
|
||||
{Position: 0, ArtistMBID: a, CreditedName: "2Pac", JoinPhrase: " feat. "},
|
||||
{Position: 1, ArtistMBID: b, CreditedName: "Snoop Dogg"},
|
||||
})
|
||||
|
||||
got, err := si.GetCredits([]string{rec})
|
||||
if err != nil {
|
||||
t.Fatalf("GetCredits: %v", err)
|
||||
}
|
||||
|
||||
parts := got[rec]
|
||||
if len(parts) != 2 {
|
||||
t.Fatalf("parts = %d, want 2", len(parts))
|
||||
}
|
||||
|
||||
var rendered strings.Builder
|
||||
for _, p := range parts {
|
||||
rendered.WriteString(p.CreditedName)
|
||||
rendered.WriteString(p.JoinPhrase)
|
||||
}
|
||||
|
||||
if rendered.String() != "2Pac feat. Snoop Dogg" {
|
||||
t.Errorf("rendered = %q, want %q", rendered.String(), "2Pac feat. Snoop Dogg")
|
||||
}
|
||||
|
||||
// Dashed on the way out: a blob reaching the frontend is sixteen
|
||||
// bytes of mojibake, and nothing above mbid.go speaks that.
|
||||
if parts[0].ArtistMBID != a {
|
||||
t.Errorf("artist mbid = %q, want %q", parts[0].ArtistMBID, a)
|
||||
}
|
||||
}
|
||||
|
||||
// TestGetCreditsOmitsSingleArtist: absence is the common case and means
|
||||
// "nothing to decompose", so the caller renders its existing one link.
|
||||
func TestGetCreditsOmitsSingleArtist(t *testing.T) {
|
||||
db := database.NewTestDB(t)
|
||||
si := NewSearchIndex(db, nil, nil, testLogger())
|
||||
|
||||
got, err := si.GetCredits([]string{testMBID("untagged"), ""})
|
||||
if err != nil {
|
||||
t.Fatalf("GetCredits: %v", err)
|
||||
}
|
||||
|
||||
if len(got) != 0 {
|
||||
t.Errorf("got %d credits, want none", len(got))
|
||||
}
|
||||
}
|
||||
|
||||
// TestGetCreditsBatches: the lookup is asked about whole tracklists, so
|
||||
// it must not build one statement per row or one SQLite refuses to
|
||||
// parse.
|
||||
func TestGetCreditsBatches(t *testing.T) {
|
||||
db := database.NewTestDB(t)
|
||||
si := NewSearchIndex(db, nil, nil, testLogger())
|
||||
|
||||
mbids := make([]string, 0, creditLookupBatch*2+7)
|
||||
for i := range creditLookupBatch*2 + 7 {
|
||||
mbids = append(mbids, testMBID(fmt.Sprintf("batch-%d", i)))
|
||||
}
|
||||
|
||||
// One real credit somewhere past the first batch boundary.
|
||||
seedCredit(t, db, mbids[creditLookupBatch+3], 9, []CreditPart{
|
||||
{Position: 0, ArtistMBID: testMBID("a"), CreditedName: "A", JoinPhrase: " & "},
|
||||
{Position: 1, ArtistMBID: testMBID("b"), CreditedName: "B"},
|
||||
})
|
||||
|
||||
got, err := si.GetCredits(mbids)
|
||||
if err != nil {
|
||||
t.Fatalf("GetCredits: %v", err)
|
||||
}
|
||||
|
||||
if len(got[mbids[creditLookupBatch+3]]) != 2 {
|
||||
t.Errorf("a credit past the first batch boundary was not returned")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,637 @@
|
||||
//go:build indexbuild
|
||||
|
||||
package explore
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bufio"
|
||||
"compress/bzip2"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"path"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Multi-artist credits, from the core MusicBrainz dump.
|
||||
//
|
||||
// A credit is ordered parts and the credit *string* is derived from
|
||||
// them; MusicBrainz's own artist_credit.name is a cached render. What
|
||||
// this pass extracts is the decomposition: for each catalog recording
|
||||
// and release group whose credit names more than one artist, the
|
||||
// credited artists in order, each with the name *as credited* and the
|
||||
// join phrase that follows it. See artist_credit_part.sql for why that
|
||||
// is stored rather than derived, and why nothing may reconstruct a
|
||||
// credit by searching a name inside a credit string.
|
||||
//
|
||||
// It is a separate dump from everything else here, and it has to be.
|
||||
// The canonical dump this importer already streams gives artist_mbids
|
||||
// (an ordered list) and artist_credit_name (the *rendered* string) --
|
||||
// no join phrases, and no per-artist as-credited names. Splitting the
|
||||
// rendered string using canonical artist names fails on exactly the
|
||||
// credits that matter: measured on a real library, 21% of multi-artist
|
||||
// credits name an artist differently from the artist's own name
|
||||
// ("Snoop Dogg" credited on a track by "Snoop Doggy Dogg"), so the
|
||||
// substring is simply not there. The JSON dumps were checked too and
|
||||
// cover 153,691 recordings of ~35M, with zero overlap against a real
|
||||
// library. This dump is the only source.
|
||||
//
|
||||
// Cost, measured on the 20260815 export: 7.1 GB compressed, decompressed
|
||||
// by pure-Go compress/bzip2 at ~26 MB/s uncompressed (~13.7 min for the
|
||||
// whole file, single-threaded). cmd/indexbuild is built CGO_ENABLED=0,
|
||||
// so the stdlib decompressor is what there is -- and it is fine, because
|
||||
// the 2 MB/s origin throttle dominates, as it does for every other dump
|
||||
// here.
|
||||
|
||||
const (
|
||||
// defaultMBDumpBaseURL is the core MusicBrainz export. Only
|
||||
// mbdump.tar.bz2 is fetched; the other tarballs there hold data this
|
||||
// app has no use for.
|
||||
defaultMBDumpBaseURL = "https://data.metabrainz.org/pub/musicbrainz/data/fullexport/"
|
||||
)
|
||||
|
||||
var (
|
||||
mbdumpDirRe = regexp.MustCompile(`^\d{8}-\d+$`)
|
||||
mbdumpFileRe = regexp.MustCompile(`^mbdump\.tar\.bz2$`)
|
||||
|
||||
// ErrDumpShape is returned when a dump member does not have the
|
||||
// columns this code was written against. It is deliberately fatal:
|
||||
// reading the wrong column silently produces a catalog whose credits
|
||||
// are subtly wrong, which is far worse than a failed build.
|
||||
ErrDumpShape = errors.New("musicbrainz dump member has an unexpected shape")
|
||||
)
|
||||
|
||||
// Column positions in the Postgres COPY output, verified against the
|
||||
// 20260815 export. There is no header row to read them from, so they
|
||||
// are asserted instead -- see checkShape.
|
||||
const (
|
||||
artistColID = 0
|
||||
artistColGID = 1
|
||||
artistColMin = 2
|
||||
|
||||
creditColID = 0
|
||||
creditColArtistCount = 2
|
||||
creditColMin = 3
|
||||
|
||||
partColCredit = 0
|
||||
partColPosition = 1
|
||||
partColArtist = 2
|
||||
partColName = 3
|
||||
partColJoin = 4
|
||||
partColMin = 5
|
||||
|
||||
// recording and release_group share a layout in the columns this
|
||||
// pass reads: id, gid, name, artist_credit, ...
|
||||
entityColGID = 1
|
||||
entityColCredit = 3
|
||||
entityColMin = 4
|
||||
)
|
||||
|
||||
// creditPart is one credited artist within a credit.
|
||||
type creditPart struct {
|
||||
position int
|
||||
artistID int32
|
||||
name string
|
||||
join string
|
||||
}
|
||||
|
||||
// creditScan is what one pass over the dump collects.
|
||||
type creditScan struct {
|
||||
// artistGIDs maps an artist row id to its MBID. artist_credit_name
|
||||
// references artists by row id, and the tar orders `artist` before
|
||||
// it, so this is complete by the time it is read.
|
||||
artistGIDs map[int32]uuid16
|
||||
|
||||
// multiCredits are the credit ids naming more than one artist, from
|
||||
// artist_credit.artist_count. Taking the count from the dump rather
|
||||
// than counting parts means a credit can be rejected before its
|
||||
// parts are stored.
|
||||
multiCredits map[int32]struct{}
|
||||
|
||||
// parts are the decompositions of multiCredits, keyed by credit id.
|
||||
parts map[int32][]creditPart
|
||||
|
||||
// refs maps a kept catalog entity to its credit. Only entities in
|
||||
// explore_index and only multi-artist credits: everything else is
|
||||
// already described by explore_index's own artist_name/artist_mbid.
|
||||
refs map[uuid16]int32
|
||||
|
||||
// used are the credits some ref actually points at, which is a small
|
||||
// fraction of multiCredits -- the catalog keeps ~1.8M entities of
|
||||
// MusicBrainz's tens of millions.
|
||||
used map[int32]struct{}
|
||||
|
||||
skippedUnknownArtist int
|
||||
}
|
||||
|
||||
// creditsImportDoneKey marks in explore_index_meta that the credit pass
|
||||
// has run against the current catalog.
|
||||
//
|
||||
// It is its own marker rather than part of the import's stage state for
|
||||
// a resume reason: the credit pass runs *after* the catalog is
|
||||
// assembled, and a failure in it must not send the next run back
|
||||
// through the ~205 GB it just finished. Marking separately means a
|
||||
// retry retries only this.
|
||||
const creditsImportDoneKey = "credits_import_done"
|
||||
|
||||
// ensureArtistCredits runs the credit pass unless it has already run
|
||||
// against this catalog, reporting whether it newly populated them.
|
||||
//
|
||||
// Called from both of run's paths -- the full import and the resume
|
||||
// that finds the rows already assembled -- and from the maintenance
|
||||
// entry point below, since a catalog built before credits existed is
|
||||
// otherwise never offered a chance to gain them: the index job picks
|
||||
// its mode from the index's own state, and a complete import means
|
||||
// "refresh", which never enters run() at all.
|
||||
//
|
||||
// The return value is what tells the job there is something new worth
|
||||
// publishing. A refresh otherwise reports "changed" only when the
|
||||
// listens series advanced, so credits would sit in the CI database and
|
||||
// never reach an artifact.
|
||||
func (imp *dumpImporter) ensureArtistCredits(ctx context.Context) bool {
|
||||
if imp.si.hasMeta(creditsImportDoneKey) {
|
||||
return false
|
||||
}
|
||||
|
||||
url, err := discoverDumpFile(
|
||||
ctx, imp.httpClient, imp.mbdumpBaseURL, mbdumpDirRe, mbdumpFileRe,
|
||||
)
|
||||
if err != nil {
|
||||
imp.logger.Warn("credit import: could not find the dump", "error", err)
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
if err := imp.importArtistCredits(ctx, url); err != nil {
|
||||
// A catalog without credits is the catalog this app shipped
|
||||
// before them: every credit falls back to its single artist.
|
||||
// That is worth far less than failing an import that otherwise
|
||||
// succeeded.
|
||||
imp.logger.Warn("credit import: failed", "error", err)
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
imp.si.setMeta(creditsImportDoneKey, "1")
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
// EnsureArtistCredits tops up the credit tables outside a full import.
|
||||
//
|
||||
// It exists because the index job's modes are decided from the index's
|
||||
// own state: a cache holding a completed import chooses `refresh`,
|
||||
// which folds in incremental listens and never enters the dump
|
||||
// importer. Without this, a catalog built before the credit pass
|
||||
// existed could only gain credits from a `rebuild` -- and a rebuild
|
||||
// re-downloads ~205 GB to reproduce rows it already has, to add
|
||||
// something that costs 7 GB on its own.
|
||||
//
|
||||
// Reports whether credits were newly populated, so the caller knows
|
||||
// there is a new artifact worth publishing.
|
||||
func (e *Service) EnsureArtistCredits(ctx context.Context) bool {
|
||||
imp, err := newDumpImporter(e.index, e.lb)
|
||||
if err != nil {
|
||||
e.index.logger.Warn("credit import: could not start", "error", err)
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
return imp.ensureArtistCredits(ctx)
|
||||
}
|
||||
|
||||
// importArtistCredits streams the core MusicBrainz dump and fills
|
||||
// artist_credit_part and artist_credit_ref for the entities the catalog
|
||||
// kept.
|
||||
//
|
||||
// It runs after assembleIndex because it asks explore_index which
|
||||
// entities those are: the popularity filter decides what is worth
|
||||
// carrying credits for, and asking the table rather than the kept sets
|
||||
// means this stays correct if that filter changes.
|
||||
func (imp *dumpImporter) importArtistCredits(ctx context.Context, url string) error {
|
||||
kept, err := imp.keptEntityMBIDs(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if len(kept) == 0 {
|
||||
imp.logger.Warn("credit import: no catalog entities, skipping")
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
imp.logger.Info("credit import: starting", "url", url, "entities", len(kept))
|
||||
imp.logJob("Streaming MusicBrainz dump for artist credits")
|
||||
|
||||
scan, err := imp.scanCreditDump(ctx, url, kept)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
imp.logger.Info("credit import: scanned",
|
||||
"multiArtistCredits", len(scan.multiCredits),
|
||||
"entitiesWithMultiArtistCredit", len(scan.refs),
|
||||
"creditsUsed", len(scan.used),
|
||||
)
|
||||
|
||||
return imp.writeCredits(ctx, scan)
|
||||
}
|
||||
|
||||
// keptEntityMBIDs is every recording and release group in the catalog.
|
||||
// Artists are excluded: an artist is not credited to a credit.
|
||||
func (imp *dumpImporter) keptEntityMBIDs(ctx context.Context) (map[uuid16]struct{}, error) {
|
||||
rows, err := imp.si.db.QueryContextWith(ctx,
|
||||
`SELECT mbid FROM explore_index
|
||||
WHERE entity_type IN (2 /* release_group */, 3 /* recording */)`,
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("credit import: read catalog entities: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
out := make(map[uuid16]struct{})
|
||||
|
||||
for rows.Next() {
|
||||
var raw []byte
|
||||
|
||||
if err := rows.Scan(&raw); err != nil {
|
||||
return nil, fmt.Errorf("credit import: scan mbid: %w", err)
|
||||
}
|
||||
|
||||
if len(raw) != len(uuid16{}) {
|
||||
continue
|
||||
}
|
||||
|
||||
var id uuid16
|
||||
|
||||
copy(id[:], raw)
|
||||
|
||||
out[id] = struct{}{}
|
||||
}
|
||||
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, fmt.Errorf("credit import: read catalog entities: %w", err)
|
||||
}
|
||||
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// scanCreditDump makes one sequential pass over mbdump.tar.bz2.
|
||||
//
|
||||
// The tar's members are alphabetical, which is what makes a single pass
|
||||
// possible without buffering the big ones: `artist` and
|
||||
// `artist_credit_name` both arrive before `recording` and
|
||||
// `release_group`, so by the time an entity names a credit, that
|
||||
// credit's parts and their artists' MBIDs are already known and the
|
||||
// entity can be resolved and dropped. 35M recording rows are never
|
||||
// held.
|
||||
//
|
||||
// The order is not depended on blindly: an entity naming a credit that
|
||||
// has not been seen is counted and reported rather than silently
|
||||
// producing an empty catalog, which is what a reordered export would
|
||||
// otherwise look like.
|
||||
func (imp *dumpImporter) scanCreditDump(
|
||||
ctx context.Context, url string, kept map[uuid16]struct{},
|
||||
) (*creditScan, error) {
|
||||
stream := imp.openDumpStream(ctx, url, 0)
|
||||
|
||||
defer func() { _ = stream.Close() }()
|
||||
|
||||
return imp.scanCreditTar(
|
||||
ctx,
|
||||
tar.NewReader(bzip2.NewReader(bufio.NewReaderSize(stream, 1<<20))),
|
||||
kept,
|
||||
)
|
||||
}
|
||||
|
||||
// scanCreditTar is the parse, separated from the fetch so it can be
|
||||
// driven by a tar built in a test. compress/bzip2 is decompress-only,
|
||||
// so a test cannot produce the real container.
|
||||
func (imp *dumpImporter) scanCreditTar(
|
||||
ctx context.Context, tr *tar.Reader, kept map[uuid16]struct{},
|
||||
) (*creditScan, error) {
|
||||
scan := &creditScan{
|
||||
artistGIDs: make(map[int32]uuid16),
|
||||
multiCredits: make(map[int32]struct{}),
|
||||
parts: make(map[int32][]creditPart),
|
||||
refs: make(map[uuid16]int32),
|
||||
used: make(map[int32]struct{}),
|
||||
}
|
||||
|
||||
for {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
hdr, err := tr.Next()
|
||||
if errors.Is(err, io.EOF) {
|
||||
break
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("credit import: tar: %w", err)
|
||||
}
|
||||
|
||||
if hdr.Typeflag != tar.TypeReg {
|
||||
continue
|
||||
}
|
||||
|
||||
done, err := imp.scanCreditMember(ctx, hdr.Name, tr, kept, scan)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if done {
|
||||
// Everything this pass needs has been read; the rest of the
|
||||
// tarball is other entities' data and decompressing it would
|
||||
// cost minutes for nothing.
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
if scan.skippedUnknownArtist > 0 {
|
||||
imp.logger.Warn("credit import: credits dropped for unknown artists",
|
||||
"count", scan.skippedUnknownArtist,
|
||||
)
|
||||
}
|
||||
|
||||
return scan, nil
|
||||
}
|
||||
|
||||
// scanCreditMember dispatches one tar member, reporting whether the
|
||||
// pass has everything it needs.
|
||||
func (imp *dumpImporter) scanCreditMember(
|
||||
ctx context.Context, name string, r io.Reader,
|
||||
kept map[uuid16]struct{}, scan *creditScan,
|
||||
) (bool, error) {
|
||||
switch path.Base(name) {
|
||||
case "artist":
|
||||
return false, imp.scanArtists(ctx, r, scan)
|
||||
case "artist_credit":
|
||||
return false, imp.scanCredits(ctx, r, scan)
|
||||
case "artist_credit_name":
|
||||
return false, imp.scanCreditParts(ctx, r, scan)
|
||||
case "recording", "release_group":
|
||||
if err := imp.scanCreditedEntities(ctx, r, kept, scan); err != nil {
|
||||
return false, err
|
||||
}
|
||||
|
||||
// release_group sorts after recording, so the pass is complete
|
||||
// once it has been read.
|
||||
return path.Base(name) == "release_group", nil
|
||||
default:
|
||||
return false, nil
|
||||
}
|
||||
}
|
||||
|
||||
// scanArtists records every artist's MBID by row id.
|
||||
func (imp *dumpImporter) scanArtists(
|
||||
ctx context.Context, r io.Reader, scan *creditScan,
|
||||
) error {
|
||||
return scanTSV(ctx, r, artistColMin, "artist", func(fields []string) error {
|
||||
id, ok := parseInt32(fields[artistColID])
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
|
||||
var gid uuid16
|
||||
|
||||
if !parseUUID(fields[artistColGID], gid[:]) {
|
||||
return fmt.Errorf("%w: artist.gid is not a UUID: %q",
|
||||
ErrDumpShape, truncate(fields[artistColGID]))
|
||||
}
|
||||
|
||||
scan.artistGIDs[id] = gid
|
||||
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// scanCredits records which credits name more than one artist.
|
||||
func (imp *dumpImporter) scanCredits(
|
||||
ctx context.Context, r io.Reader, scan *creditScan,
|
||||
) error {
|
||||
return scanTSV(ctx, r, creditColMin, "artist_credit", func(fields []string) error {
|
||||
id, ok := parseInt32(fields[creditColID])
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
|
||||
count, ok := parseInt32(fields[creditColArtistCount])
|
||||
if !ok {
|
||||
return fmt.Errorf("%w: artist_credit.artist_count is not a number: %q",
|
||||
ErrDumpShape, truncate(fields[creditColArtistCount]))
|
||||
}
|
||||
|
||||
if count > 1 {
|
||||
scan.multiCredits[id] = struct{}{}
|
||||
}
|
||||
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// scanCreditParts records the decomposition of every multi-artist
|
||||
// credit.
|
||||
func (imp *dumpImporter) scanCreditParts(
|
||||
ctx context.Context, r io.Reader, scan *creditScan,
|
||||
) error {
|
||||
return scanTSV(ctx, r, partColMin, "artist_credit_name", func(fields []string) error {
|
||||
credit, ok := parseInt32(fields[partColCredit])
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
|
||||
if _, multi := scan.multiCredits[credit]; !multi {
|
||||
return nil
|
||||
}
|
||||
|
||||
position, ok := parseInt32(fields[partColPosition])
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
|
||||
artist, ok := parseInt32(fields[partColArtist])
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
|
||||
scan.parts[credit] = append(scan.parts[credit], creditPart{
|
||||
position: int(position),
|
||||
artistID: artist,
|
||||
name: fields[partColName],
|
||||
join: fields[partColJoin],
|
||||
})
|
||||
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// scanCreditedEntities resolves recordings and release groups against
|
||||
// the catalog, keeping only those the catalog holds and whose credit
|
||||
// names more than one artist.
|
||||
func (imp *dumpImporter) scanCreditedEntities(
|
||||
ctx context.Context, r io.Reader, kept map[uuid16]struct{}, scan *creditScan,
|
||||
) error {
|
||||
return scanTSV(ctx, r, entityColMin, "recording/release_group",
|
||||
func(fields []string) error {
|
||||
var gid uuid16
|
||||
|
||||
if !parseUUID(fields[entityColGID], gid[:]) {
|
||||
return fmt.Errorf("%w: entity gid is not a UUID: %q",
|
||||
ErrDumpShape, truncate(fields[entityColGID]))
|
||||
}
|
||||
|
||||
if _, want := kept[gid]; !want {
|
||||
return nil
|
||||
}
|
||||
|
||||
credit, ok := parseInt32(fields[entityColCredit])
|
||||
if !ok {
|
||||
return fmt.Errorf("%w: entity artist_credit is not a number: %q",
|
||||
ErrDumpShape, truncate(fields[entityColCredit]))
|
||||
}
|
||||
|
||||
if _, multi := scan.multiCredits[credit]; !multi {
|
||||
return nil
|
||||
}
|
||||
|
||||
scan.refs[gid] = credit
|
||||
scan.used[credit] = struct{}{}
|
||||
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// scanTSV reads Postgres COPY output a line at a time, unescaping each
|
||||
// field and handing the row to fn.
|
||||
//
|
||||
// The shape is asserted on the first row rather than trusted: this dump
|
||||
// has no header, so a column that moved would otherwise be read as a
|
||||
// neighbouring one and produce a catalog that is quietly wrong.
|
||||
func scanTSV(
|
||||
ctx context.Context, r io.Reader, minCols int, member string,
|
||||
fn func(fields []string) error,
|
||||
) error {
|
||||
sc := bufio.NewScanner(r)
|
||||
sc.Buffer(make([]byte, 0, 1<<20), 1<<24)
|
||||
|
||||
checked := false
|
||||
rows := 0
|
||||
|
||||
for sc.Scan() {
|
||||
rows++
|
||||
|
||||
if rows%(1<<20) == 0 {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
line := sc.Text()
|
||||
if line == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
fields := strings.Split(line, "\t")
|
||||
if len(fields) < minCols {
|
||||
if !checked {
|
||||
return fmt.Errorf("%w: %s has %d columns, need at least %d",
|
||||
ErrDumpShape, member, len(fields), minCols)
|
||||
}
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
checked = true
|
||||
|
||||
for i := range fields {
|
||||
fields[i] = unescapeCopy(fields[i])
|
||||
}
|
||||
|
||||
if err := fn(fields); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
if err := sc.Err(); err != nil {
|
||||
return fmt.Errorf("credit import: read %s: %w", member, err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// unescapeCopy undoes Postgres COPY's text escaping. A NULL (\N) is
|
||||
// returned as an empty string: every field this pass reads is either a
|
||||
// number it will reject or a name whose absence means the same as
|
||||
// empty.
|
||||
func unescapeCopy(s string) string {
|
||||
if s == `\N` {
|
||||
return ""
|
||||
}
|
||||
|
||||
if !strings.ContainsRune(s, '\\') {
|
||||
return s
|
||||
}
|
||||
|
||||
var b strings.Builder
|
||||
|
||||
b.Grow(len(s))
|
||||
|
||||
for i := 0; i < len(s); i++ {
|
||||
if s[i] != '\\' || i+1 >= len(s) {
|
||||
b.WriteByte(s[i])
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
i++
|
||||
|
||||
switch s[i] {
|
||||
case 'n':
|
||||
b.WriteByte('\n')
|
||||
case 't':
|
||||
b.WriteByte('\t')
|
||||
case 'r':
|
||||
b.WriteByte('\r')
|
||||
case 'b':
|
||||
b.WriteByte('\b')
|
||||
case 'f':
|
||||
b.WriteByte('\f')
|
||||
case 'v':
|
||||
b.WriteByte('\v')
|
||||
case '\\':
|
||||
b.WriteByte('\\')
|
||||
default:
|
||||
b.WriteByte('\\')
|
||||
b.WriteByte(s[i])
|
||||
}
|
||||
}
|
||||
|
||||
return b.String()
|
||||
}
|
||||
|
||||
func parseInt32(s string) (int32, bool) {
|
||||
n, err := strconv.ParseInt(s, 10, 32)
|
||||
if err != nil {
|
||||
return 0, false
|
||||
}
|
||||
|
||||
return int32(n), true
|
||||
}
|
||||
|
||||
// truncate bounds an error message built from dump data, which is
|
||||
// attacker-free but can be long.
|
||||
func truncate(s string) string {
|
||||
const limit = 64
|
||||
|
||||
if len(s) <= limit {
|
||||
return s
|
||||
}
|
||||
|
||||
return s[:limit] + "..."
|
||||
}
|
||||
@@ -0,0 +1,406 @@
|
||||
//go:build indexbuild
|
||||
|
||||
package explore
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"yellowjacket/backend/database"
|
||||
)
|
||||
|
||||
// tarOf builds an uncompressed tar of the named members, in the order
|
||||
// given. Order is the point of several of these tests: the real dump's
|
||||
// members are alphabetical, which is what lets one pass resolve an
|
||||
// entity's credit without buffering 35M recordings.
|
||||
func tarOf(t *testing.T, members ...[2]string) *tar.Reader {
|
||||
t.Helper()
|
||||
|
||||
var buf bytes.Buffer
|
||||
|
||||
tw := tar.NewWriter(&buf)
|
||||
|
||||
for _, m := range members {
|
||||
body := []byte(m[1])
|
||||
|
||||
if err := tw.WriteHeader(&tar.Header{
|
||||
Name: "mbdump/" + m[0],
|
||||
Mode: 0o644,
|
||||
Size: int64(len(body)),
|
||||
Typeflag: tar.TypeReg,
|
||||
}); err != nil {
|
||||
t.Fatalf("tar header: %v", err)
|
||||
}
|
||||
|
||||
if _, err := tw.Write(body); err != nil {
|
||||
t.Fatalf("tar write: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
if err := tw.Close(); err != nil {
|
||||
t.Fatalf("tar close: %v", err)
|
||||
}
|
||||
|
||||
return tar.NewReader(&buf)
|
||||
}
|
||||
|
||||
func tsv(rows ...[]string) string {
|
||||
var b strings.Builder
|
||||
|
||||
for _, r := range rows {
|
||||
b.WriteString(strings.Join(r, "\t"))
|
||||
b.WriteByte('\n')
|
||||
}
|
||||
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// mustMBID is testMBID in the packed form the catalog stores.
|
||||
func mustMBID(label string) uuid16 {
|
||||
var u uuid16
|
||||
|
||||
if !parseUUID(testMBID(label), u[:]) {
|
||||
panic("testMBID did not produce a UUID for " + label)
|
||||
}
|
||||
|
||||
return u
|
||||
}
|
||||
|
||||
// The two artists of the worked example, and the entities they credit.
|
||||
var (
|
||||
creditRecMBID = mustMBID("recording-1")
|
||||
creditRGMBID = mustMBID("release-group-1")
|
||||
)
|
||||
|
||||
// sampleDump is the shape verified against the 20260815 export:
|
||||
// artist(id, gid, ...), artist_credit(id, name, artist_count, ...),
|
||||
// artist_credit_name(credit, position, artist, name, join_phrase),
|
||||
// recording/release_group(id, gid, name, artist_credit, ...).
|
||||
func sampleDump(t *testing.T) *tar.Reader {
|
||||
t.Helper()
|
||||
|
||||
return tarOf(t,
|
||||
[2]string{"artist", tsv(
|
||||
[]string{"11", testMBID("artist-a"), "Snoop Doggy Dogg", "Snoop Doggy Dogg"},
|
||||
[]string{"22", testMBID("artist-b"), "2Pac", "2Pac"},
|
||||
)},
|
||||
[2]string{"artist_credit", tsv(
|
||||
[]string{"900", "2Pac feat. Snoop Dogg", "2", "1", "", "0", ""},
|
||||
[]string{"901", "Solo Artist", "1", "1", "", "0", ""},
|
||||
)},
|
||||
[2]string{"artist_credit_name", tsv(
|
||||
// Deliberately out of position order: the dump is not
|
||||
// obliged to emit them sorted and the credit's meaning is
|
||||
// the order, not the file's.
|
||||
[]string{"900", "1", "11", "Snoop Dogg", ""},
|
||||
[]string{"900", "0", "22", "2Pac", " feat. "},
|
||||
[]string{"901", "0", "11", "Solo Artist", ""},
|
||||
)},
|
||||
[2]string{"recording", tsv(
|
||||
[]string{"1", testMBID("recording-1"), "Some Song", "900", "180000"},
|
||||
[]string{"2", testMBID("not-kept"), "Other", "900", "1"},
|
||||
[]string{"3", testMBID("solo"), "Solo", "901", "1"},
|
||||
)},
|
||||
[2]string{"release_group", tsv(
|
||||
[]string{"5", testMBID("release-group-1"), "Some Album", "900", "1"},
|
||||
)},
|
||||
)
|
||||
}
|
||||
|
||||
func creditTestImporter(t *testing.T) *dumpImporter {
|
||||
t.Helper()
|
||||
|
||||
db := database.NewTestDB(t)
|
||||
|
||||
return &dumpImporter{
|
||||
si: NewSearchIndex(db, nil, nil, testLogger()),
|
||||
logger: testLogger(),
|
||||
}
|
||||
}
|
||||
|
||||
// TestScanCreditDumpDecomposes is the worked example end to end: the
|
||||
// credit's parts come back in position order, with the *credited*
|
||||
// names and the join phrase between them.
|
||||
func TestScanCreditDumpDecomposes(t *testing.T) {
|
||||
imp := creditTestImporter(t)
|
||||
|
||||
kept := map[uuid16]struct{}{
|
||||
creditRecMBID: {},
|
||||
creditRGMBID: {},
|
||||
}
|
||||
|
||||
scan, err := imp.scanCreditTar(context.Background(), sampleDump(t), kept)
|
||||
if err != nil {
|
||||
t.Fatalf("scan: %v", err)
|
||||
}
|
||||
|
||||
if got := len(scan.refs); got != 2 {
|
||||
t.Fatalf("refs = %d, want 2 (the recording and the release group)", got)
|
||||
}
|
||||
|
||||
if scan.refs[creditRecMBID] != 900 {
|
||||
t.Errorf("recording credit = %d, want 900", scan.refs[creditRecMBID])
|
||||
}
|
||||
|
||||
parts := scan.parts[900]
|
||||
if len(parts) != 2 {
|
||||
t.Fatalf("parts = %d, want 2", len(parts))
|
||||
}
|
||||
|
||||
// Sorting happens on write, so assert the pieces are all present
|
||||
// and let the render test below check the order.
|
||||
byPos := map[int]creditPart{}
|
||||
for _, p := range parts {
|
||||
byPos[p.position] = p
|
||||
}
|
||||
|
||||
if byPos[0].name != "2Pac" || byPos[0].join != " feat. " {
|
||||
t.Errorf("position 0 = %q/%q, want \"2Pac\"/\" feat. \"",
|
||||
byPos[0].name, byPos[0].join)
|
||||
}
|
||||
|
||||
// The credited name, not the artist's own name: this is the whole
|
||||
// reason credited_name is stored per row.
|
||||
if byPos[1].name != "Snoop Dogg" {
|
||||
t.Errorf("position 1 credited name = %q, want \"Snoop Dogg\"", byPos[1].name)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSingleArtistCreditsAreNotStored: a one-artist credit is already
|
||||
// described by explore_index's artist_name/artist_mbid, and storing it
|
||||
// would roughly triple the table to say nothing new.
|
||||
func TestSingleArtistCreditsAreNotStored(t *testing.T) {
|
||||
imp := creditTestImporter(t)
|
||||
|
||||
solo := mustMBID("solo")
|
||||
kept := map[uuid16]struct{}{solo: {}}
|
||||
|
||||
scan, err := imp.scanCreditTar(context.Background(), sampleDump(t), kept)
|
||||
if err != nil {
|
||||
t.Fatalf("scan: %v", err)
|
||||
}
|
||||
|
||||
if len(scan.refs) != 0 {
|
||||
t.Fatalf("a single-artist credit was referenced: %v", scan.refs)
|
||||
}
|
||||
|
||||
if _, ok := scan.multiCredits[901]; ok {
|
||||
t.Error("credit 901 has artist_count 1 and should not be multi")
|
||||
}
|
||||
}
|
||||
|
||||
// TestOnlyKeptEntitiesAreReferenced: the catalog's popularity filter
|
||||
// decides what is worth carrying credits for, and an entity outside it
|
||||
// must not produce a row pointing at nothing.
|
||||
func TestOnlyKeptEntitiesAreReferenced(t *testing.T) {
|
||||
imp := creditTestImporter(t)
|
||||
|
||||
kept := map[uuid16]struct{}{creditRecMBID: {}}
|
||||
|
||||
scan, err := imp.scanCreditTar(context.Background(), sampleDump(t), kept)
|
||||
if err != nil {
|
||||
t.Fatalf("scan: %v", err)
|
||||
}
|
||||
|
||||
if _, ok := scan.refs[mustMBID("not-kept")]; ok {
|
||||
t.Error("an entity outside the catalog was referenced")
|
||||
}
|
||||
|
||||
if len(scan.used) != 1 {
|
||||
t.Errorf("used credits = %d, want 1", len(scan.used))
|
||||
}
|
||||
}
|
||||
|
||||
// TestWriteCreditsRoundTrips checks what the frontend will actually
|
||||
// read: parts in position order, dashed MBIDs out of the 16 raw bytes,
|
||||
// and a rendered credit that reassembles to the tagged string.
|
||||
func TestWriteCreditsRoundTrips(t *testing.T) {
|
||||
imp := creditTestImporter(t)
|
||||
|
||||
kept := map[uuid16]struct{}{creditRecMBID: {}, creditRGMBID: {}}
|
||||
|
||||
scan, err := imp.scanCreditTar(context.Background(), sampleDump(t), kept)
|
||||
if err != nil {
|
||||
t.Fatalf("scan: %v", err)
|
||||
}
|
||||
|
||||
if err := imp.writeCredits(context.Background(), scan); err != nil {
|
||||
t.Fatalf("writeCredits: %v", err)
|
||||
}
|
||||
|
||||
rows, err := imp.si.db.QueryContext(
|
||||
`SELECT p.position, p.artist_mbid, p.credited_name, p.join_phrase
|
||||
FROM artist_credit_ref r
|
||||
JOIN artist_credit_part p ON p.credit_id = r.credit_id
|
||||
WHERE r.mbid = ?
|
||||
ORDER BY p.position`,
|
||||
creditRecMBID[:],
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("query: %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
var rendered strings.Builder
|
||||
|
||||
names := []string{}
|
||||
|
||||
for rows.Next() {
|
||||
var (
|
||||
pos int
|
||||
mbid []byte
|
||||
name string
|
||||
join string
|
||||
)
|
||||
|
||||
if err := rows.Scan(&pos, &mbid, &name, &join); err != nil {
|
||||
t.Fatalf("scan row: %v", err)
|
||||
}
|
||||
|
||||
if len(mbid) != 16 {
|
||||
t.Fatalf("artist_mbid is %d bytes, want 16", len(mbid))
|
||||
}
|
||||
|
||||
names = append(names, name)
|
||||
|
||||
rendered.WriteString(name)
|
||||
rendered.WriteString(join)
|
||||
}
|
||||
|
||||
if err := rows.Err(); err != nil {
|
||||
t.Fatalf("rows: %v", err)
|
||||
}
|
||||
|
||||
// Concatenation is the contract: names in order, join phrases
|
||||
// between them, and no searching a name inside a credit string.
|
||||
if got := rendered.String(); got != "2Pac feat. Snoop Dogg" {
|
||||
t.Errorf("rendered credit = %q, want %q", got, "2Pac feat. Snoop Dogg")
|
||||
}
|
||||
|
||||
if len(names) != 2 || names[0] != "2Pac" {
|
||||
t.Errorf("parts came back out of position order: %v", names)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCreditRefsNeverDangle: a ref whose parts were not stored renders
|
||||
// as a credit with no artists at all, which is worse than the
|
||||
// single-artist fallback it replaced.
|
||||
func TestCreditRefsNeverDangle(t *testing.T) {
|
||||
imp := creditTestImporter(t)
|
||||
|
||||
kept := map[uuid16]struct{}{creditRecMBID: {}}
|
||||
|
||||
scan, err := imp.scanCreditTar(context.Background(), sampleDump(t), kept)
|
||||
if err != nil {
|
||||
t.Fatalf("scan: %v", err)
|
||||
}
|
||||
|
||||
// An artist the dump never named: the credit cannot be navigated to
|
||||
// and must be dropped whole, taking its ref with it.
|
||||
scan.artistGIDs = map[int32]uuid16{}
|
||||
|
||||
if err := imp.writeCredits(context.Background(), scan); err != nil {
|
||||
t.Fatalf("writeCredits: %v", err)
|
||||
}
|
||||
|
||||
var refs, parts int
|
||||
|
||||
if err := imp.si.db.QueryRowWriter(
|
||||
"SELECT COUNT(*) FROM artist_credit_ref",
|
||||
).Scan(&refs); err != nil {
|
||||
t.Fatalf("count refs: %v", err)
|
||||
}
|
||||
|
||||
if err := imp.si.db.QueryRowWriter(
|
||||
"SELECT COUNT(*) FROM artist_credit_part",
|
||||
).Scan(&parts); err != nil {
|
||||
t.Fatalf("count parts: %v", err)
|
||||
}
|
||||
|
||||
if refs != 0 || parts != 0 {
|
||||
t.Fatalf("refs=%d parts=%d, want 0/0 when the artists are unknown", refs, parts)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCreditDumpShapeIsAsserted: the dump has no header row, so a
|
||||
// column that moved would be read as its neighbour and produce a
|
||||
// catalog that is quietly wrong. Loud is the requirement.
|
||||
func TestCreditDumpShapeIsAsserted(t *testing.T) {
|
||||
imp := creditTestImporter(t)
|
||||
|
||||
short := tarOf(t, [2]string{"artist", tsv([]string{"11", "only-two-columns"})})
|
||||
|
||||
_, err := imp.scanCreditTar(context.Background(), short, map[uuid16]struct{}{})
|
||||
if err == nil {
|
||||
t.Fatal("a member with a non-UUID gid was accepted")
|
||||
}
|
||||
|
||||
if !errors.Is(err, ErrDumpShape) {
|
||||
t.Errorf("error = %v, want ErrDumpShape", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestUnescapeCopy covers Postgres COPY's text escaping, which reaches
|
||||
// artist names routinely -- a tab or backslash in a name would
|
||||
// otherwise shift every field after it.
|
||||
func TestUnescapeCopy(t *testing.T) {
|
||||
tests := []struct{ in, want string }{
|
||||
{`plain`, `plain`},
|
||||
{`\N`, ``},
|
||||
{`a\tb`, "a\tb"},
|
||||
{`a\nb`, "a\nb"},
|
||||
{`back\\slash`, `back\slash`},
|
||||
{`AC\/DC`, `AC\/DC`},
|
||||
{`trailing\`, `trailing\`},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
if got := unescapeCopy(tt.in); got != tt.want {
|
||||
t.Errorf("unescapeCopy(%q) = %q, want %q", tt.in, got, tt.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestEnsureArtistCreditsIsIdempotent pins what the index job depends
|
||||
// on to decide whether to publish.
|
||||
//
|
||||
// The pass runs on every mode, including the `refresh` that a complete
|
||||
// catalog always chooses — so it must be free when there is nothing to
|
||||
// do, and it must say so. A `true` here republishes the artifact; a
|
||||
// `true` on every run would republish an identical one weekly, and a
|
||||
// permanent `false` would mean a catalog that never gains credits at
|
||||
// all.
|
||||
func TestEnsureArtistCreditsIsIdempotent(t *testing.T) {
|
||||
imp := creditTestImporter(t)
|
||||
|
||||
// The marker is what "already done" means; with it set, the pass
|
||||
// must not reach the network or report a change.
|
||||
imp.si.setMeta(creditsImportDoneKey, "1")
|
||||
|
||||
if imp.ensureArtistCredits(context.Background()) {
|
||||
t.Fatal("a second run reported new credits; the artifact would republish forever")
|
||||
}
|
||||
}
|
||||
|
||||
// TestEnsureArtistCreditsReportsFailureAsNoChange: a dump that cannot be
|
||||
// reached leaves the catalog exactly as it was, and must not claim
|
||||
// otherwise — publishing on it would ship an artifact with no credits
|
||||
// and mark the work done.
|
||||
func TestEnsureArtistCreditsReportsFailureAsNoChange(t *testing.T) {
|
||||
imp := creditTestImporter(t)
|
||||
imp.httpClient = newDumpHTTPClient()
|
||||
imp.mbdumpBaseURL = "http://127.0.0.1:1/nonexistent/"
|
||||
|
||||
if imp.ensureArtistCredits(context.Background()) {
|
||||
t.Fatal("an unreachable dump reported new credits")
|
||||
}
|
||||
|
||||
if imp.si.hasMeta(creditsImportDoneKey) {
|
||||
t.Error("a failed pass marked itself done; it would never retry")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,164 @@
|
||||
//go:build indexbuild
|
||||
|
||||
package explore
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"sort"
|
||||
)
|
||||
|
||||
// writeCredits persists the scanned decompositions.
|
||||
//
|
||||
// Only credits some catalog entity actually points at are written: the
|
||||
// dump has millions of multi-artist credits and the catalog keeps ~1.8M
|
||||
// entities, so storing every credit would be most of a table nothing
|
||||
// can reach.
|
||||
//
|
||||
// The two tables are written in one transaction, because a ref pointing
|
||||
// at parts that are not there renders as a credit with no artists --
|
||||
// worse than the single-artist fallback it replaced.
|
||||
func (imp *dumpImporter) writeCredits(ctx context.Context, scan *creditScan) error {
|
||||
tx, err := imp.si.db.BeginTx()
|
||||
if err != nil {
|
||||
return fmt.Errorf("credit import: begin: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = tx.Rollback() }()
|
||||
|
||||
// A rebuild replaces the previous pass wholesale. These are Cache
|
||||
// tables derived entirely from the dump, so there is nothing to
|
||||
// merge and a stale row is a wrong credit.
|
||||
for _, table := range []string{"artist_credit_part", "artist_credit_ref"} {
|
||||
if _, err := tx.ExecContext(ctx, "DELETE FROM "+table); err != nil {
|
||||
return fmt.Errorf("credit import: clear %s: %w", table, err)
|
||||
}
|
||||
}
|
||||
|
||||
written, err := imp.writeCreditParts(ctx, tx, scan)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
refs, err := imp.writeCreditRefs(ctx, tx, scan, written)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if err := tx.Commit(); err != nil {
|
||||
return fmt.Errorf("credit import: commit: %w", err)
|
||||
}
|
||||
|
||||
imp.logger.Info("credit import: complete",
|
||||
"credits", len(written),
|
||||
"refs", refs,
|
||||
)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// writeCreditParts inserts the parts of every used credit and returns
|
||||
// the set of credits that were actually stored.
|
||||
//
|
||||
// A credit is stored whole or not at all. If any of its artists has no
|
||||
// MBID -- which should not happen, the dump being self-consistent, but
|
||||
// would leave a part that cannot be navigated to -- the credit is
|
||||
// dropped and the entity falls back to explore_index's single artist,
|
||||
// which is a worse answer rather than a broken one.
|
||||
func (imp *dumpImporter) writeCreditParts(
|
||||
ctx context.Context, tx *sql.Tx, scan *creditScan,
|
||||
) (map[int32]struct{}, error) {
|
||||
stmt, err := tx.PrepareContext(ctx,
|
||||
`INSERT INTO artist_credit_part
|
||||
(credit_id, position, artist_mbid, credited_name, join_phrase)
|
||||
VALUES (?, ?, ?, ?, ?)`,
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("credit import: prepare part insert: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = stmt.Close() }()
|
||||
|
||||
written := make(map[int32]struct{}, len(scan.used))
|
||||
|
||||
for credit := range scan.used {
|
||||
parts := scan.parts[credit]
|
||||
if len(parts) < 2 {
|
||||
// artist_credit said more than one artist and
|
||||
// artist_credit_name did not deliver them. Nothing to
|
||||
// decompose, so leave the entity to its single artist.
|
||||
continue
|
||||
}
|
||||
|
||||
// Position order is the credit's meaning, and the dump is not
|
||||
// obliged to emit it sorted.
|
||||
sort.Slice(parts, func(i, j int) bool {
|
||||
return parts[i].position < parts[j].position
|
||||
})
|
||||
|
||||
resolved := make([][]any, 0, len(parts))
|
||||
ok := true
|
||||
|
||||
for _, part := range parts {
|
||||
gid, found := scan.artistGIDs[part.artistID]
|
||||
if !found {
|
||||
scan.skippedUnknownArtist++
|
||||
ok = false
|
||||
|
||||
break
|
||||
}
|
||||
|
||||
resolved = append(resolved, []any{
|
||||
credit, part.position, gid[:], part.name, part.join,
|
||||
})
|
||||
}
|
||||
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
|
||||
for _, args := range resolved {
|
||||
if _, err := stmt.ExecContext(ctx, args...); err != nil {
|
||||
return nil, fmt.Errorf("credit import: insert part: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
written[credit] = struct{}{}
|
||||
}
|
||||
|
||||
return written, nil
|
||||
}
|
||||
|
||||
// writeCreditRefs points each kept entity at its credit, skipping any
|
||||
// whose credit was not stored so a ref never dangles.
|
||||
func (imp *dumpImporter) writeCreditRefs(
|
||||
ctx context.Context, tx *sql.Tx, scan *creditScan, written map[int32]struct{},
|
||||
) (int, error) {
|
||||
stmt, err := tx.PrepareContext(ctx,
|
||||
"INSERT OR REPLACE INTO artist_credit_ref (mbid, credit_id) VALUES (?, ?)",
|
||||
)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("credit import: prepare ref insert: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = stmt.Close() }()
|
||||
|
||||
count := 0
|
||||
|
||||
for mbid, credit := range scan.refs {
|
||||
if _, stored := written[credit]; !stored {
|
||||
continue
|
||||
}
|
||||
|
||||
id := mbid
|
||||
|
||||
if _, err := stmt.ExecContext(ctx, id[:], credit); err != nil {
|
||||
return 0, fmt.Errorf("credit import: insert ref: %w", err)
|
||||
}
|
||||
|
||||
count++
|
||||
}
|
||||
|
||||
return count, nil
|
||||
}
|
||||
@@ -104,6 +104,7 @@ type dumpImporter struct {
|
||||
|
||||
canonicalBaseURL string
|
||||
listensBaseURL string
|
||||
mbdumpBaseURL string
|
||||
|
||||
// Disk safety floors (fields so tests can relax them).
|
||||
minStartFreeBytes uint64
|
||||
@@ -144,6 +145,7 @@ func newDumpImporter(si *SearchIndex, lb *ListenBrainzClient) (*dumpImporter, er
|
||||
stagingDir: stagingDir,
|
||||
canonicalBaseURL: defaultCanonicalBaseURL,
|
||||
listensBaseURL: defaultListensBaseURL,
|
||||
mbdumpBaseURL: defaultMBDumpBaseURL,
|
||||
minStartFreeBytes: dumpMinStartFreeBytes,
|
||||
abortFreeBytes: dumpAbortFreeBytes,
|
||||
}, nil
|
||||
@@ -171,6 +173,7 @@ func (imp *dumpImporter) run(ctx context.Context) error {
|
||||
// Fast path: rows already assembled, only patch passes remain.
|
||||
if state.Stage == dumpStageAssembled {
|
||||
imp.si.MarkReadyIfPopulated()
|
||||
imp.ensureArtistCredits(ctx)
|
||||
imp.runPatchPasses(ctx)
|
||||
|
||||
if err := ctx.Err(); err != nil {
|
||||
@@ -305,6 +308,11 @@ func (imp *dumpImporter) run(ctx context.Context) error {
|
||||
imp.si.MarkReadyIfPopulated()
|
||||
imp.si.refreshStatusCounts()
|
||||
|
||||
// Multi-artist credits, from a different dump. After the catalog,
|
||||
// because it asks explore_index which entities are worth carrying
|
||||
// credits for.
|
||||
imp.ensureArtistCredits(ctx)
|
||||
|
||||
// Stage 4: API patch passes (idempotent).
|
||||
imp.runPatchPasses(ctx)
|
||||
|
||||
|
||||
+14
-1
@@ -177,12 +177,25 @@ func run(o opts) error {
|
||||
|
||||
complete := svc.IndexImportComplete() && !errors.Is(err, errIncomplete)
|
||||
|
||||
// Credits are maintenance, not part of any one mode. They come from
|
||||
// a different dump, they are keyed on entities the catalog already
|
||||
// holds, and a catalog built before the pass existed would otherwise
|
||||
// only gain them from a rebuild — which re-downloads ~205 GB to
|
||||
// re-derive rows it already has. Skipped when the import is not
|
||||
// complete, because there is nothing to key them against yet.
|
||||
creditsAdded := false
|
||||
if complete {
|
||||
creditsAdded = svc.EnsureArtistCredits(context.Background())
|
||||
}
|
||||
|
||||
// "Changed" means there is something new worth publishing, so it is
|
||||
// only ever true for a finished import: a build stamps the listens
|
||||
// series early, long before its rows are assembled, and reporting a
|
||||
// change off that would be a lie about a half-built index.
|
||||
changed := complete &&
|
||||
(svc.IndexBaselineSeries() != seriesBefore || chosen != modeRefresh)
|
||||
(svc.IndexBaselineSeries() != seriesBefore ||
|
||||
chosen != modeRefresh ||
|
||||
creditsAdded)
|
||||
|
||||
report(logger, svc, chosen, complete, changed)
|
||||
|
||||
|
||||
@@ -53,10 +53,19 @@ func TestRetireLibraryTables(t *testing.T) {
|
||||
CREATE TABLE recordings (id INTEGER PRIMARY KEY, title TEXT);
|
||||
`)
|
||||
|
||||
// The symptom, before the repair: the schema cannot be applied over
|
||||
// a table whose shape has moved on.
|
||||
if _, err := database.NewDB(logger); err == nil {
|
||||
t.Fatal("expected the stale shape to fail to open; it did not")
|
||||
// This used to assert the symptom -- that the schema cannot be
|
||||
// applied over a table whose shape has moved on -- because at the
|
||||
// time nothing repaired it and only this job did. The app-side
|
||||
// repair (backend/database/staleshape.go) now retires a stale
|
||||
// non-authored table before applySchema meets it, so opening
|
||||
// succeeds and the symptom no longer reproduces from here.
|
||||
//
|
||||
// That does not make retireLibraryTables redundant, and the rest of
|
||||
// this test is why: the app-side repair only removes what is *stale*,
|
||||
// while this database wants its library half gone entirely, healthy
|
||||
// or not, because nothing here scans, plays or authors.
|
||||
if _, err := database.NewDB(logger); err != nil {
|
||||
t.Fatalf("the app-side repair should have opened this: %v", err)
|
||||
}
|
||||
|
||||
if err := retireLibraryTables(context.Background(), logger); err != nil {
|
||||
|
||||
@@ -207,6 +207,26 @@ func createSchema(db *sql.DB) error {
|
||||
key TEXT PRIMARY KEY,
|
||||
value TEXT NOT NULL
|
||||
)`,
|
||||
// Multi-artist credits. Shipped as their own tables rather than
|
||||
// as an explore_index column because a credit is a variable
|
||||
// number of ordered parts, and because credits are *shared* --
|
||||
// an album's tracks by one artist reference one credit, which is
|
||||
// what keeps this to a few hundred thousand rows.
|
||||
//
|
||||
// An importer that predates these reads an artifact without
|
||||
// them; artifactHasCredits is what asks.
|
||||
`CREATE TABLE core.artist_credit_part (
|
||||
credit_id INTEGER NOT NULL,
|
||||
position INTEGER NOT NULL,
|
||||
artist_mbid BLOB NOT NULL,
|
||||
credited_name TEXT NOT NULL,
|
||||
join_phrase TEXT NOT NULL DEFAULT '',
|
||||
PRIMARY KEY (credit_id, position)
|
||||
) WITHOUT ROWID`,
|
||||
`CREATE TABLE core.artist_credit_ref (
|
||||
mbid BLOB NOT NULL PRIMARY KEY,
|
||||
credit_id INTEGER NOT NULL
|
||||
) WITHOUT ROWID`,
|
||||
}
|
||||
|
||||
for _, stmt := range stmts {
|
||||
@@ -283,6 +303,63 @@ func copyRows(db *sql.DB, artists, perArtistRGs, perArtistRecs int) error {
|
||||
fmt.Printf(" %-15s %d\n", sel.label+":", n)
|
||||
}
|
||||
|
||||
return copyCredits(db)
|
||||
}
|
||||
|
||||
// copyCredits ships the credit decomposition for the entities that made
|
||||
// it into the artifact, and only those.
|
||||
//
|
||||
// The refs go first and the parts follow *from* the refs, so a credit is
|
||||
// carried only if something in the artifact points at it. The source
|
||||
// index holds credits for every catalog entity, while the artifact is a
|
||||
// windowed subset -- copying all of them would carry a large table most
|
||||
// of which nothing in the artifact can reach.
|
||||
//
|
||||
// A source index built before the credit pass simply has no rows here,
|
||||
// which is not an error: the artifact then carries the tables empty, and
|
||||
// every credit falls back to its single artist exactly as before.
|
||||
func copyCredits(db *sql.DB) error {
|
||||
// Asked, not assumed. A source index built before the credit pass
|
||||
// has no such table, and "no such table" would fail an export whose
|
||||
// catalog is otherwise complete.
|
||||
for _, table := range []string{"artist_credit_ref", "artist_credit_part"} {
|
||||
var n int
|
||||
|
||||
if err := db.QueryRow(
|
||||
`SELECT COUNT(*) FROM main.sqlite_master
|
||||
WHERE type = 'table' AND name = ?`, table,
|
||||
).Scan(&n); err != nil {
|
||||
return fmt.Errorf("probe %s: %w", table, err)
|
||||
}
|
||||
|
||||
if n == 0 {
|
||||
fmt.Printf(" %-15s none in source\n", "credits:")
|
||||
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
refs, err := insertSelect(db, `
|
||||
INSERT INTO core.artist_credit_ref (mbid, credit_id)
|
||||
SELECT r.mbid, r.credit_id
|
||||
FROM main.artist_credit_ref r
|
||||
WHERE r.mbid IN (SELECT mbid FROM core.explore_index)`)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
parts, err := insertSelect(db, `
|
||||
INSERT INTO core.artist_credit_part
|
||||
(credit_id, position, artist_mbid, credited_name, join_phrase)
|
||||
SELECT p.credit_id, p.position, p.artist_mbid, p.credited_name, p.join_phrase
|
||||
FROM main.artist_credit_part p
|
||||
WHERE p.credit_id IN (SELECT credit_id FROM core.artist_credit_ref)`)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
fmt.Printf(" %-15s %d refs, %d parts\n", "credits:", refs, parts)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -12,6 +12,7 @@ export {
|
||||
|
||||
export type {
|
||||
AlbumCompleteFunc,
|
||||
CreditPart,
|
||||
IndexStatus,
|
||||
LBSimilarArtist,
|
||||
LBTopRecording,
|
||||
|
||||
@@ -7,6 +7,23 @@
|
||||
*/
|
||||
export type AlbumCompleteFunc = any;
|
||||
|
||||
/**
|
||||
* CreditPart is one credited artist within a credit, in credit order.
|
||||
*
|
||||
* CreditedName is the name *as credited*, which is not the artist's own
|
||||
* name: MusicBrainz credits "Snoop Dogg" on a track by the artist
|
||||
* called "Snoop Doggy Dogg". Display uses it; navigation uses
|
||||
* ArtistMBID. JoinPhrase is the literal connector that follows this
|
||||
* part, so a credit renders by concatenation and never by searching a
|
||||
* name inside a credit string.
|
||||
*/
|
||||
export interface CreditPart {
|
||||
"position": number;
|
||||
"artistMbid": string;
|
||||
"creditedName": string;
|
||||
"joinPhrase": string;
|
||||
}
|
||||
|
||||
/**
|
||||
* IndexStatus is the full index build status, exposed to the frontend.
|
||||
*/
|
||||
|
||||
@@ -225,6 +225,18 @@ export function GetCandidateThumbnail(releaseMBID: string, releaseGroupMBID: str
|
||||
return $Call.ByID(1946932424, releaseMBID, releaseGroupMBID);
|
||||
}
|
||||
|
||||
/**
|
||||
* GetCredits is the bound form: the frontend asks for a tracklist's
|
||||
* worth of MBIDs at once rather than one per row.
|
||||
*
|
||||
* Batched for the reason every other per-row backend question here is:
|
||||
* asking on hover or on render turns a list into N IPC round trips, and
|
||||
* this one is asked about every row of every list in the app.
|
||||
*/
|
||||
export function GetCredits(mbids: string[] | null): $CancellablePromise<{ [_ in string]?: $models.CreditPart[] | null } | null> {
|
||||
return $Call.ByID(225964099, mbids);
|
||||
}
|
||||
|
||||
/**
|
||||
* GetExploreShelves builds the page Explore shows before a query.
|
||||
*
|
||||
|
||||
@@ -52,7 +52,8 @@ import {
|
||||
} from '@utils/context-menu-controller.js';
|
||||
import type { ContextMenuHost } from '@utils/context-menu-controller.js';
|
||||
import { FavoritesController } from '@store/controllers/favorites-controller';
|
||||
import { artistLink, exploreLinkStyles } from '../../utils/explore-link';
|
||||
import { creditLink, exploreLinkStyles } from '../../utils/explore-link';
|
||||
import { creditStore } from '@store/credit-store';
|
||||
import {
|
||||
createAlbumArtDragImage,
|
||||
createDragImage,
|
||||
@@ -425,8 +426,18 @@ export class CoverGrid
|
||||
* Lifecycle
|
||||
* ==================================================================== */
|
||||
|
||||
/** Unsubscribes the credit-arrival repaint. */
|
||||
private creditsUnsub?: () => void;
|
||||
|
||||
override connectedCallback() {
|
||||
super.connectedCallback();
|
||||
|
||||
this.creditsUnsub = creditStore.subscribe(() => {
|
||||
this.requestUpdate();
|
||||
// Two virtualizers when the grid is split; both draw rows.
|
||||
this.renderRoot?.querySelectorAll('lit-virtualizer')
|
||||
.forEach((v) => (v as unknown as { requestUpdate(): void }).requestUpdate());
|
||||
});
|
||||
this.restoreSortPreferences();
|
||||
this.loadAlbums();
|
||||
|
||||
@@ -441,6 +452,8 @@ export class CoverGrid
|
||||
|
||||
override disconnectedCallback() {
|
||||
super.disconnectedCallback();
|
||||
this.creditsUnsub?.();
|
||||
this.creditsUnsub = undefined;
|
||||
|
||||
this.removeEventListener(
|
||||
'error',
|
||||
@@ -1820,7 +1833,7 @@ export class CoverGrid
|
||||
class="artist-name"
|
||||
title="${album.ArtistName}"
|
||||
>
|
||||
${artistLink(album.ArtistName, album.ArtistMBID ?? '')}
|
||||
${creditLink(creditStore.credits(album.MBID), album.ArtistName, album.ArtistMBID ?? '')}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -20,7 +20,8 @@ type MBRelease = explore.MBRelease;
|
||||
type MBTrack = explore.MBTrack;
|
||||
import { exploreCache } from '../../store/explore-cache';
|
||||
import { libraryStore } from '../../store/library-store';
|
||||
import { artistLink, exploreLinkStyles } from '../../utils/explore-link';
|
||||
import { creditLink, exploreLinkStyles } from '../../utils/explore-link';
|
||||
import { creditStore } from '@store/credit-store';
|
||||
import { describeError } from '../../utils/describe-error';
|
||||
import { EventsOn } from '@runtime/runtime';
|
||||
import { Events } from '../../events';
|
||||
@@ -704,8 +705,15 @@ export class ExploreAlbumDetails extends LitElement implements ContextMenuHost {
|
||||
* BrowseReleases fetch never signals readiness. */
|
||||
private releasesFallbackTimer?: number;
|
||||
|
||||
/** Unsubscribes the credit-arrival repaint. */
|
||||
private creditsUnsub?: () => void;
|
||||
|
||||
override connectedCallback() {
|
||||
super.connectedCallback();
|
||||
|
||||
this.creditsUnsub = creditStore.subscribe(() => {
|
||||
this.requestUpdate();
|
||||
});
|
||||
if (this.releaseGroupMBID || this.localAlbumId) {
|
||||
void this.loadAllData();
|
||||
}
|
||||
@@ -760,6 +768,8 @@ export class ExploreAlbumDetails extends LitElement implements ContextMenuHost {
|
||||
|
||||
override disconnectedCallback() {
|
||||
super.disconnectedCallback();
|
||||
this.creditsUnsub?.();
|
||||
this.creditsUnsub = undefined;
|
||||
this.downloadUnsub?.();
|
||||
this.downloadUnsub = null;
|
||||
this.unsubReleasesReady?.();
|
||||
@@ -2850,7 +2860,11 @@ export class ExploreAlbumDetails extends LitElement implements ContextMenuHost {
|
||||
return html`
|
||||
${artist
|
||||
? html`<div class="album-artist">
|
||||
${artistLink(artist, artistMbid)}
|
||||
${creditLink(
|
||||
creditStore.credits(this.releaseGroupMBID),
|
||||
artist,
|
||||
artistMbid,
|
||||
)}
|
||||
</div>`
|
||||
: nothing}
|
||||
${metaParts.length > 0
|
||||
|
||||
@@ -16,7 +16,8 @@ import { exploreCache, ARTIST_IMAGE_CACHE_LIMIT } from '../../store/explore-cach
|
||||
import { queueStore } from '../../store/queue-store';
|
||||
import { notificationStore } from '../../store/notification-store';
|
||||
import '../notifications/inline-notice';
|
||||
import { artistLink, trackLink, exploreLinkStyles } from '../../utils/explore-link';
|
||||
import { creditLink, trackLink, exploreLinkStyles } from '../../utils/explore-link';
|
||||
import { creditStore } from '@store/credit-store';
|
||||
import { describeError } from '../../utils/describe-error';
|
||||
import '@awesome.me/webawesome/dist/components/icon/icon.js';
|
||||
import '../library-status-indicator/library-status-indicator.js';
|
||||
@@ -774,6 +775,14 @@ export class ExploreView extends ViewLifecycleMixin(LitElement) implements Conte
|
||||
}
|
||||
|
||||
protected override onViewActivate(): void {
|
||||
// A cached primary view, so this is torn down on the way out
|
||||
// rather than on disconnect — which never fires here.
|
||||
this.whileActive(
|
||||
creditStore.subscribe(() => {
|
||||
this.requestUpdate();
|
||||
}),
|
||||
);
|
||||
|
||||
// Fetched on arrival rather than on connect: this is a cached
|
||||
// primary view, created and warmed at startup, so a fetch there
|
||||
// is three catalog queries every user pays for whether or not
|
||||
@@ -2193,7 +2202,7 @@ export class ExploreView extends ViewLifecycleMixin(LitElement) implements Conte
|
||||
<div class="album-title" title="${rg.title}">
|
||||
${rg.title}
|
||||
</div>
|
||||
<div class="album-artist">${artistLink(rg.artistCredit, rg.artistMbid ?? '')}</div>
|
||||
<div class="album-artist">${creditLink(creditStore.credits(rg.mbid), rg.artistCredit, rg.artistMbid ?? '')}</div>
|
||||
<div class="album-meta">
|
||||
<div class="album-meta-text">
|
||||
${rg.primaryType
|
||||
@@ -2255,7 +2264,7 @@ export class ExploreView extends ViewLifecycleMixin(LitElement) implements Conte
|
||||
${trackLink(r.title, r.releaseName ?? '', r.releaseGroupMbid ?? '', r.mbid)}
|
||||
</div>
|
||||
<div class="track-artist">
|
||||
${artistLink(r.artistCredit, r.artistMbid ?? '')}
|
||||
${creditLink(creditStore.credits(r.mbid), r.artistCredit, r.artistMbid ?? '')}
|
||||
</div>
|
||||
</div>
|
||||
<div class="track-meta">
|
||||
|
||||
@@ -5,11 +5,12 @@ import '../audio-player/controls/player-controls';
|
||||
import '../audio-player/seekbar/seek-bar';
|
||||
import '../audio-player/volume-control/volume-control';
|
||||
import {
|
||||
artistLink,
|
||||
creditLink,
|
||||
albumLink,
|
||||
exploreLinkStyles,
|
||||
} from '@utils/explore-link';
|
||||
import { PlayerController } from '@store/controllers/player-controller';
|
||||
import { creditStore } from '@store/credit-store';
|
||||
import { FavoritesController } from '@store/controllers/favorites-controller';
|
||||
import { designTokens } from '../../styles/tokens.css';
|
||||
import { srOnly } from '../../styles/sr-only.css';
|
||||
@@ -36,6 +37,22 @@ import { srOnly } from '../../styles/sr-only.css';
|
||||
@customElement('now-playing-view')
|
||||
export class NowPlayingView extends LitElement {
|
||||
private player = new PlayerController(this);
|
||||
|
||||
/** Unsubscribes the credit-arrival repaint. */
|
||||
private creditsUnsub?: () => void;
|
||||
|
||||
override connectedCallback(): void {
|
||||
super.connectedCallback();
|
||||
// Credits arrive after the track does, so the name this view is
|
||||
// already showing has to be re-rendered when they land.
|
||||
this.creditsUnsub = creditStore.subscribe(() => this.requestUpdate());
|
||||
}
|
||||
|
||||
override disconnectedCallback(): void {
|
||||
super.disconnectedCallback();
|
||||
this.creditsUnsub?.();
|
||||
this.creditsUnsub = undefined;
|
||||
}
|
||||
private favCtrl = new FavoritesController(this);
|
||||
|
||||
static override styles = [designTokens, srOnly, exploreLinkStyles, css`
|
||||
@@ -262,7 +279,11 @@ export class NowPlayingView extends LitElement {
|
||||
${track.title || track.fileName}
|
||||
</h2>
|
||||
<p class="artist">
|
||||
${artistLink(track.artist, track.artistMbid)}
|
||||
${creditLink(
|
||||
creditStore.credits(track.recordingMbid),
|
||||
track.artist,
|
||||
track.artistMbid,
|
||||
)}
|
||||
</p>
|
||||
${track.album
|
||||
? html`<p class="album">
|
||||
|
||||
@@ -4,7 +4,7 @@ import '@awesome.me/webawesome/dist/components/icon/icon.js';
|
||||
import '@awesome.me/webawesome/dist/components/popup/popup.js';
|
||||
import type WaPopup from '@awesome.me/webawesome/dist/components/popup/popup.js';
|
||||
import {
|
||||
artistLink,
|
||||
creditLink,
|
||||
trackLink,
|
||||
exploreLinkStyles,
|
||||
} from '@utils/explore-link';
|
||||
@@ -14,6 +14,7 @@ import {
|
||||
navigateToQueueSource,
|
||||
} from '@utils/queue-source-link';
|
||||
import { PlayerController } from '@store/controllers/player-controller';
|
||||
import { creditStore } from '@store/credit-store';
|
||||
import { QueueController } from '@store/controllers/queue-controller';
|
||||
import { FavoritesController } from '@store/controllers/favorites-controller';
|
||||
import { designTokens } from '../../styles/tokens.css';
|
||||
@@ -325,6 +326,9 @@ export class NowPlaying extends LitElement {
|
||||
}
|
||||
`];
|
||||
|
||||
/** Unsubscribes the credit-arrival repaint. */
|
||||
private creditsUnsub?: () => void;
|
||||
|
||||
override connectedCallback() {
|
||||
super.connectedCallback();
|
||||
this.loadScrollMode();
|
||||
@@ -341,10 +345,21 @@ export class NowPlaying extends LitElement {
|
||||
this.geometryDirty = true;
|
||||
this.requestUpdate();
|
||||
});
|
||||
|
||||
// A credit arriving changes the rendered text, and the marquee
|
||||
// measures that text — so this is a geometry change, not just a
|
||||
// repaint. Saying so is what stops the bar scrolling to the
|
||||
// old width.
|
||||
this.creditsUnsub = creditStore.subscribe(() => {
|
||||
this.geometryDirty = true;
|
||||
this.requestUpdate();
|
||||
});
|
||||
}
|
||||
|
||||
override disconnectedCallback() {
|
||||
super.disconnectedCallback();
|
||||
this.creditsUnsub?.();
|
||||
this.creditsUnsub = undefined;
|
||||
// A drag interrupted by the bar going away still has to clean up.
|
||||
this.attachDragListeners(false);
|
||||
window.removeEventListener(SCROLL_CHANGE_EVENT, this.handleScrollModeEvent);
|
||||
@@ -483,7 +498,7 @@ export class NowPlaying extends LitElement {
|
||||
@mouseleave=${this.handleArtistMouseLeave}
|
||||
@transitionend=${() => this.onScrollCycleEnd('artist')}
|
||||
>
|
||||
<span class="scroll-content">${artistLink(track.artist, track.artistMbid) || 'Unknown Artist'}</span>
|
||||
<span class="scroll-content">${creditLink(creditStore.credits(track.recordingMbid), track.artist, track.artistMbid) || 'Unknown Artist'}</span>
|
||||
</span>
|
||||
${describeQueueSource(this.queue.source)
|
||||
? html`
|
||||
|
||||
@@ -24,6 +24,7 @@ import type * as playlist from '@go/playlist/models.js';
|
||||
import { EventsOn } from '@runtime/runtime';
|
||||
import { Events } from '../../events';
|
||||
import { queueStore } from '@store/queue-store';
|
||||
import { creditStore } from '@store/credit-store';
|
||||
import { PlayerController } from '@store/controllers/player-controller';
|
||||
import { SearchController } from '@store/controllers/search-controller';
|
||||
import { SelectionController } from '@utils/selection-controller';
|
||||
@@ -61,7 +62,8 @@ import '@components/duplicate-tracks-dialog/duplicate-tracks-dialog.js';
|
||||
import type { DuplicateTracksDialog } from '@components/duplicate-tracks-dialog/duplicate-tracks-dialog.js';
|
||||
import { formatMilliseconds } from '@utils/time';
|
||||
import {
|
||||
artistLink,
|
||||
creditLink,
|
||||
creditText,
|
||||
albumLink,
|
||||
trackLink,
|
||||
exploreLinkStyles,
|
||||
@@ -106,6 +108,9 @@ export class PlaylistDetails
|
||||
* rather than guessed. Without the hint the flow layout's 100 px
|
||||
* default drives constant scroll-error correction, which reads as
|
||||
* the list jumping under the pointer. */
|
||||
/** Unsubscribes the credit-arrival repaint. */
|
||||
private creditsUnsub?: () => void;
|
||||
|
||||
@query('lit-virtualizer')
|
||||
private virtualizer?: LitVirtualizer;
|
||||
|
||||
@@ -224,6 +229,14 @@ export class PlaylistDetails
|
||||
|
||||
override connectedCallback() {
|
||||
super.connectedCallback();
|
||||
|
||||
// Credits arrive after the rows that asked for them, and a
|
||||
// virtualizer repaints from its *own* properties — a host
|
||||
// update alone leaves the rows exactly as they were.
|
||||
this.creditsUnsub = creditStore.subscribe(() => {
|
||||
this.requestUpdate();
|
||||
this.virtualizer?.requestUpdate();
|
||||
});
|
||||
this.loadTracks();
|
||||
|
||||
this.tracksChangedCleanup = EventsOn(
|
||||
@@ -248,6 +261,8 @@ export class PlaylistDetails
|
||||
|
||||
override disconnectedCallback() {
|
||||
super.disconnectedCallback();
|
||||
this.creditsUnsub?.();
|
||||
this.creditsUnsub = undefined;
|
||||
|
||||
if (this.tracksChangedCleanup) {
|
||||
this.tracksChangedCleanup();
|
||||
@@ -1575,7 +1590,7 @@ export class PlaylistDetails
|
||||
: nothing}
|
||||
</div>
|
||||
<span class="cell col-title" title="${track.Title || track.FilePath}">${trackLink(track.Title, track.Album, track.ReleaseGroupMBID, track.RecordingMBID, undefined, track.Artist) || track.FilePath}</span>
|
||||
<span class="cell col-artist" title="${track.Artist}">${artistLink(track.Artist, track.ArtistMBID)}</span>
|
||||
<span class="cell col-artist" title="${creditText(creditStore.credits(track.RecordingMBID), track.Artist)}">${creditLink(creditStore.credits(track.RecordingMBID), track.Artist, track.ArtistMBID)}</span>
|
||||
<span class="cell col-album" title="${track.Album}">${albumLink(track.Album, track.ReleaseGroupMBID, undefined, track.Artist)}</span>
|
||||
<span class="cell col-duration">${formatMilliseconds(track.Duration)}</span>`}
|
||||
</div>
|
||||
|
||||
@@ -12,6 +12,7 @@ import '@awesome.me/webawesome/dist/components/popup/popup.js';
|
||||
import type WaPopup from '@awesome.me/webawesome/dist/components/popup/popup.js';
|
||||
import '@awesome.me/webawesome/dist/components/dropdown-item/dropdown-item.js';
|
||||
import { QueueController } from '@store/controllers/queue-controller';
|
||||
import { creditStore } from '@store/credit-store';
|
||||
import {
|
||||
describeQueueSource,
|
||||
isQueueSourceNavigable,
|
||||
@@ -55,7 +56,7 @@ import { tracksByFilePath } from '@utils/track-index.js';
|
||||
import type { TrackDetails } from '@components/track-details/track-details.js';
|
||||
import type { CoverArtUrls } from '@components/track-details/track-details.js';
|
||||
import {
|
||||
artistLink,
|
||||
creditLink,
|
||||
trackLink,
|
||||
exploreLinkStyles,
|
||||
} from '@utils/explore-link';
|
||||
@@ -108,6 +109,9 @@ export class QueuePanel
|
||||
@query('#playlist-submenu')
|
||||
private playlistSubmenuPopup!: WaPopup;
|
||||
|
||||
/** Unsubscribes the credit-arrival repaint. */
|
||||
private creditsUnsub?: () => void;
|
||||
|
||||
@query('lit-virtualizer')
|
||||
private virtualizer!: LitVirtualizer;
|
||||
|
||||
@@ -635,6 +639,14 @@ export class QueuePanel
|
||||
|
||||
override connectedCallback() {
|
||||
super.connectedCallback();
|
||||
|
||||
// Credits arrive after the rows that asked for them, and a
|
||||
// virtualizer repaints from its *own* properties — a host
|
||||
// update alone leaves the rows exactly as they were.
|
||||
this.creditsUnsub = creditStore.subscribe(() => {
|
||||
this.requestUpdate();
|
||||
this.virtualizer?.requestUpdate();
|
||||
});
|
||||
this.style.setProperty(
|
||||
'--queue-width',
|
||||
`${this.panelWidth}px`,
|
||||
@@ -671,6 +683,8 @@ export class QueuePanel
|
||||
|
||||
override disconnectedCallback() {
|
||||
super.disconnectedCallback();
|
||||
this.creditsUnsub?.();
|
||||
this.creditsUnsub = undefined;
|
||||
document.removeEventListener(
|
||||
'mousemove',
|
||||
this.handleMouseMove,
|
||||
@@ -1675,7 +1689,7 @@ export class QueuePanel
|
||||
${trackLink(title, track.album, track.releaseGroupMbid, track.recordingMbid, undefined, track.artist)}
|
||||
</span>
|
||||
<span class="track-artist" title=${artist}>
|
||||
${artistLink(track.artist, track.artistMbid) || 'Unknown Artist'}
|
||||
${creditLink(creditStore.credits(track.recordingMbid), track.artist, track.artistMbid) || 'Unknown Artist'}
|
||||
</span>
|
||||
</div>
|
||||
<button
|
||||
|
||||
@@ -15,6 +15,7 @@ import {
|
||||
import { EventsOn } from '@runtime/runtime';
|
||||
import { Events } from '../../events';
|
||||
import { queueStore } from '@store/queue-store';
|
||||
import { creditStore } from '@store/credit-store';
|
||||
import { PlayerController } from '@store/controllers/player-controller';
|
||||
import { SearchController } from '@store/controllers/search-controller';
|
||||
import { SelectionController } from '@utils/selection-controller';
|
||||
@@ -51,7 +52,8 @@ import type { CoverArtUrls } from '@components/track-details/track-details.js';
|
||||
import { libraryStore } from '@store/library-store';
|
||||
import { formatMilliseconds } from '@utils/time';
|
||||
import {
|
||||
artistLink,
|
||||
creditLink,
|
||||
creditText,
|
||||
albumLink,
|
||||
trackLink,
|
||||
exploreLinkStyles,
|
||||
@@ -136,6 +138,9 @@ export class SmartPlaylistDetails
|
||||
* rather than guessed: without the hint the flow layout's 100 px
|
||||
* default drives constant scroll-error correction, which reads as
|
||||
* the list jumping under the pointer. */
|
||||
/** Unsubscribes the credit-arrival repaint. */
|
||||
private creditsUnsub?: () => void;
|
||||
|
||||
@query('lit-virtualizer')
|
||||
private virtualizer?: LitVirtualizer;
|
||||
|
||||
@@ -609,6 +614,14 @@ export class SmartPlaylistDetails
|
||||
override connectedCallback() {
|
||||
super.connectedCallback();
|
||||
|
||||
// Credits arrive after the rows that asked for them, and a
|
||||
// virtualizer repaints from its *own* properties — a host
|
||||
// update alone leaves the rows exactly as they were.
|
||||
this.creditsUnsub = creditStore.subscribe(() => {
|
||||
this.requestUpdate();
|
||||
this.virtualizer?.requestUpdate();
|
||||
});
|
||||
|
||||
if (this.autoEdit) {
|
||||
// Skip evaluation for new playlists — go straight to editor.
|
||||
this.autoEdit = false;
|
||||
@@ -649,6 +662,8 @@ export class SmartPlaylistDetails
|
||||
|
||||
override disconnectedCallback() {
|
||||
super.disconnectedCallback();
|
||||
this.creditsUnsub?.();
|
||||
this.creditsUnsub = undefined;
|
||||
|
||||
if (this.playlistDeletedCleanup) {
|
||||
this.playlistDeletedCleanup();
|
||||
@@ -1423,7 +1438,7 @@ export class SmartPlaylistDetails
|
||||
: nothing}
|
||||
</div>
|
||||
<span class="cell col-title" title="${track.Title || track.FilePath}">${trackLink(track.Title, track.Album, track.ReleaseGroupMBID, track.RecordingMBID, undefined, track.Artist) || track.FilePath}</span>
|
||||
<span class="cell col-artist" title="${track.Artist}">${artistLink(track.Artist, track.ArtistMBID)}</span>
|
||||
<span class="cell col-artist" title="${creditText(creditStore.credits(track.RecordingMBID), track.Artist)}">${creditLink(creditStore.credits(track.RecordingMBID), track.Artist, track.ArtistMBID)}</span>
|
||||
<span class="cell col-album" title="${track.Album}">${albumLink(track.Album, track.ReleaseGroupMBID, undefined, track.Artist)}</span>
|
||||
<span class="cell col-duration">${formatMilliseconds(track.Duration)}</span>`}
|
||||
</div>
|
||||
|
||||
@@ -9,7 +9,8 @@ import {
|
||||
} from '@go/explore/service.js';
|
||||
import '../library-status-indicator/library-status-indicator.js';
|
||||
import type { LibraryStatus } from '../library-status-indicator/library-status-indicator.js';
|
||||
import { artistLink, exploreLinkStyles } from '../../utils/explore-link';
|
||||
import { creditLink, exploreLinkStyles } from '../../utils/explore-link';
|
||||
import { creditStore } from '@store/credit-store';
|
||||
import { libraryStatusFor } from '../../utils/library-status';
|
||||
import { downloadStore } from '../../store/download-store';
|
||||
|
||||
@@ -61,14 +62,23 @@ export class TopResultsRow extends LitElement {
|
||||
* the property and never updates this element. One subscription for
|
||||
* the row, not one per card.
|
||||
*/
|
||||
/** Unsubscribes the credit-arrival repaint. */
|
||||
private creditsUnsub?: () => void;
|
||||
|
||||
override connectedCallback(): void {
|
||||
super.connectedCallback();
|
||||
|
||||
this.creditsUnsub = creditStore.subscribe(() => {
|
||||
this.requestUpdate();
|
||||
});
|
||||
this.unsubRequests = downloadStore.subscribe(() =>
|
||||
this.requestUpdate(),
|
||||
);
|
||||
}
|
||||
|
||||
override disconnectedCallback(): void {
|
||||
this.creditsUnsub?.();
|
||||
this.creditsUnsub = undefined;
|
||||
this.unsubRequests?.();
|
||||
this.unsubRequests = undefined;
|
||||
super.disconnectedCallback();
|
||||
@@ -327,7 +337,7 @@ export class TopResultsRow extends LitElement {
|
||||
${artistPart || metaPart
|
||||
? html`<span class="card-subtitle"
|
||||
>${artistPart
|
||||
? artistLink(artistPart, r.artistMbid ?? '')
|
||||
? creditLink(creditStore.credits(r.mbid), artistPart, r.artistMbid ?? '')
|
||||
: nothing}${artistPart && metaPart
|
||||
? ' · '
|
||||
: ''}${metaPart}</span
|
||||
|
||||
@@ -25,6 +25,7 @@ import type { SortOption } from '@components/page-header/page-header';
|
||||
import { TrackListController } from '@store/controllers/tracklist-controller';
|
||||
import { FavoritesController } from '@store/controllers/favorites-controller';
|
||||
import { queueStore } from '@store/queue-store';
|
||||
import { creditStore } from '@store/credit-store';
|
||||
import type { QueueSource } from '@store/queue-store';
|
||||
import { LibraryController } from '@store/controllers/library-controller';
|
||||
import {
|
||||
@@ -40,6 +41,7 @@ import {
|
||||
} from './search-ranking';
|
||||
import {
|
||||
artistLink,
|
||||
creditLink,
|
||||
albumLink,
|
||||
trackLink,
|
||||
exploreLinkStyles,
|
||||
@@ -1339,6 +1341,17 @@ export class TrackList
|
||||
'shortcut:tracklist-delete',
|
||||
this.handleShortcutDelete,
|
||||
);
|
||||
|
||||
// Credits arrive after the rows that asked for them. The
|
||||
// virtualizer produces its rows from its *own* properties, so a
|
||||
// host re-render alone repaints nothing — the same reason a
|
||||
// selection change pushes requestUpdate() into it.
|
||||
this.whileActive(
|
||||
creditStore.subscribe(() => {
|
||||
this.requestUpdate();
|
||||
this.virtualizer?.requestUpdate();
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -2150,7 +2163,15 @@ export class TrackList
|
||||
if (col.id === 'trackName') {
|
||||
display = trackLink(track.TrackName, track.Album, track.ReleaseGroupMBID, track.RecordingMBID, display as any, track.ArtistName);
|
||||
} else if (col.id === 'artistName') {
|
||||
display = artistLink(track.ArtistName, track.ArtistMBID, display as any);
|
||||
// A search term highlights the *flat* credit string,
|
||||
// and mapping those spans onto decomposed parts is a
|
||||
// different problem from rendering the credit. While
|
||||
// filtering, the single link is the honest answer.
|
||||
creditStore.request(track.RecordingMBID);
|
||||
const parts = term ? undefined : creditStore.get(track.RecordingMBID);
|
||||
display = parts && parts.length > 1
|
||||
? creditLink(parts, track.ArtistName, track.ArtistMBID)
|
||||
: artistLink(track.ArtistName, track.ArtistMBID, display as any);
|
||||
} else if (col.id === 'album') {
|
||||
display = albumLink(track.Album, track.ReleaseGroupMBID, display as any, track.ArtistName);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,226 @@
|
||||
/**
|
||||
* Multi-artist credits, keyed by recording MBID.
|
||||
*
|
||||
* A credit is ordered parts and the credit *string* is derived from
|
||||
* them. This store holds the parts for entities that have more than
|
||||
* one credited artist; everything else renders the single link it
|
||||
* always did.
|
||||
*
|
||||
* Three things about it are load-bearing.
|
||||
*
|
||||
* **Absence is an answer, and it is cached as one.** The backend
|
||||
* returns nothing for a single-artist credit, which is the common case
|
||||
* by a wide margin — measured on a real library, 13% of tracks are
|
||||
* multi-artist. Caching only the hits would re-request the other 87%
|
||||
* on every render, forever, which is the same shape as the bug that
|
||||
* made `explore-album-details` ask the backend on hover. A miss is
|
||||
* stored as an empty array: *asked*, not *answered*.
|
||||
*
|
||||
* **The lookup is batched, and coalesced across callers.** Every row
|
||||
* of every tracklist asks this question, and one IPC round trip per row
|
||||
* is how a 5,000-row list becomes unusable. A virtualized list cannot
|
||||
* hand over "the whole list" either — 50,000 rows is 100 queries for
|
||||
* the ~30 on screen. So `request()` is per-row and cheap: it collects
|
||||
* into a pending set and flushes once on the next frame, which turns a
|
||||
* screenful of rows into exactly one call. `ensure()` remains for a
|
||||
* caller that genuinely has a bounded list in hand.
|
||||
*
|
||||
* **It is bounded.** A cache that grows with use is a leak with a
|
||||
* schedule; a browsing afternoon touches far more credits than a
|
||||
* screenful. The cap is entries rather than bytes because a credit is
|
||||
* a handful of short strings, unlike the art caches next door.
|
||||
*/
|
||||
|
||||
import { GetCredits } from '@go/explore/service.js';
|
||||
import type { CreditPart } from '../utils/explore-link';
|
||||
import { LRUMap } from '../utils/lru-map';
|
||||
import { compact } from '../utils/binding';
|
||||
import { registerCacheProbe } from '../utils/cache-stats';
|
||||
|
||||
/**
|
||||
* Entries retained. A credit is ~4 short strings, so this is well
|
||||
* under a megabyte — sized to comfortably exceed any single list the
|
||||
* app renders, because a cap below the visible count evicts rows that
|
||||
* are still on screen and the re-render fetches them straight back.
|
||||
*/
|
||||
export const CREDIT_CACHE_LIMIT = 20_000;
|
||||
|
||||
/** An empty parts array is the negative marker: asked, no decomposition. */
|
||||
type CachedParts = readonly CreditPart[];
|
||||
|
||||
class CreditStore {
|
||||
private cache = new LRUMap<string, CachedParts>(CREDIT_CACHE_LIMIT);
|
||||
|
||||
/** MBIDs with a request in flight, so a re-render does not refetch. */
|
||||
private inFlight = new Set<string>();
|
||||
|
||||
private listeners = new Set<() => void>();
|
||||
|
||||
/** Collected by request(), flushed as one batch on the next frame. */
|
||||
private pending = new Set<string>();
|
||||
|
||||
private flushHandle: number | null = null;
|
||||
|
||||
constructor() {
|
||||
registerCacheProbe('credits', () => ({
|
||||
entries: this.cache.size,
|
||||
chars: this.retainedChars(),
|
||||
limit: CREDIT_CACHE_LIMIT,
|
||||
}));
|
||||
}
|
||||
|
||||
/**
|
||||
* The strings actually retained, counted rather than estimated —
|
||||
* a bound that is only checkable against a guess is not checkable.
|
||||
*/
|
||||
private retainedChars(): number {
|
||||
let total = 0;
|
||||
|
||||
for (const parts of this.cache.values()) {
|
||||
for (const part of parts) {
|
||||
total +=
|
||||
part.creditedName.length +
|
||||
part.joinPhrase.length +
|
||||
part.artistMbid.length;
|
||||
}
|
||||
}
|
||||
|
||||
return total;
|
||||
}
|
||||
|
||||
/**
|
||||
* Subscribe to "some credits arrived".
|
||||
*
|
||||
* Deliberately not per-MBID: a list fetches its rows in one call and
|
||||
* re-renders once, so a fine-grained signal would buy nothing and
|
||||
* cost a listener per row.
|
||||
*/
|
||||
subscribe(fn: () => void): () => void {
|
||||
this.listeners.add(fn);
|
||||
|
||||
return () => this.listeners.delete(fn);
|
||||
}
|
||||
|
||||
/**
|
||||
* The parts for one entity, or undefined when it has not been asked
|
||||
* about yet.
|
||||
*
|
||||
* An entity with a single-artist credit returns an empty array, and
|
||||
* `creditLink` treats fewer than two parts as the fallback — so a
|
||||
* caller does not have to distinguish "not asked" from "one artist"
|
||||
* to render correctly, only to decide whether to ask.
|
||||
*/
|
||||
get(mbid: string | undefined): readonly CreditPart[] | undefined {
|
||||
if (!mbid) return undefined;
|
||||
|
||||
return this.cache.get(mbid);
|
||||
}
|
||||
|
||||
/**
|
||||
* Ask about one entity, joining whatever batch is forming.
|
||||
*
|
||||
* Safe to call from a render: it is a set insert and a scheduled
|
||||
* flush, and an entity already cached or in flight is dropped. The
|
||||
* loop it looks like it might cause does not happen — after a flush
|
||||
* every requested MBID is cached, so the re-render's requests are
|
||||
* all dropped and nothing notifies again.
|
||||
*/
|
||||
request(mbid: string | undefined): void {
|
||||
if (!mbid) return;
|
||||
if (this.cache.has(mbid)) return;
|
||||
if (this.inFlight.has(mbid)) return;
|
||||
if (this.pending.has(mbid)) return;
|
||||
|
||||
this.pending.add(mbid);
|
||||
|
||||
if (this.flushHandle !== null) return;
|
||||
|
||||
// A frame, not a microtask: the point is to collect every row a
|
||||
// virtualizer renders in this pass, and those happen across the
|
||||
// whole update, not within one microtask checkpoint.
|
||||
this.flushHandle = requestAnimationFrame(() => {
|
||||
this.flushHandle = null;
|
||||
|
||||
const batch = [...this.pending];
|
||||
|
||||
this.pending.clear();
|
||||
|
||||
void this.ensure(batch);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Ask and read in one call, for use inside a template.
|
||||
*
|
||||
* A getter with a side effect, deliberately: the alternative is
|
||||
* every call site writing `request(x)` beside `get(x)` and one of
|
||||
* them eventually forgetting, which renders a permanently
|
||||
* single-artist credit that looks exactly like an entity with one
|
||||
* artist. Making the request the same act as the read is what
|
||||
* stops the two drifting apart.
|
||||
*/
|
||||
credits(mbid: string | undefined): readonly CreditPart[] | undefined {
|
||||
this.request(mbid);
|
||||
|
||||
return this.get(mbid);
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch the credits for a list, skipping anything already known or
|
||||
* already being fetched.
|
||||
*
|
||||
* `has` rather than `get` for the membership test: probing must not
|
||||
* mark an entry recently-used, or scrolling past a row would keep
|
||||
* it alive ahead of one actually being rendered.
|
||||
*/
|
||||
async ensure(mbids: readonly (string | undefined)[]): Promise<void> {
|
||||
const wanted = new Set<string>();
|
||||
|
||||
for (const mbid of mbids) {
|
||||
if (!mbid) continue;
|
||||
if (this.cache.has(mbid)) continue;
|
||||
if (this.inFlight.has(mbid)) continue;
|
||||
|
||||
wanted.add(mbid);
|
||||
}
|
||||
|
||||
if (wanted.size === 0) return;
|
||||
|
||||
const batch = [...wanted];
|
||||
|
||||
for (const mbid of batch) this.inFlight.add(mbid);
|
||||
|
||||
try {
|
||||
const found = compact(await GetCredits(batch));
|
||||
|
||||
for (const mbid of batch) {
|
||||
// Every MBID asked for gets an entry, present or not:
|
||||
// the absent ones are the answer "one artist", and not
|
||||
// recording that is what would re-ask forever.
|
||||
this.cache.set(mbid, found[mbid] ?? []);
|
||||
}
|
||||
|
||||
this.notify();
|
||||
} catch (err) {
|
||||
// A credit is an enrichment: without it every name renders
|
||||
// as the single link it did before, which is a worse answer
|
||||
// rather than a broken one. Nothing user-facing is worth
|
||||
// interrupting for, so this stays in the console.
|
||||
console.error('Failed to load artist credits', err);
|
||||
} finally {
|
||||
for (const mbid of batch) this.inFlight.delete(mbid);
|
||||
}
|
||||
}
|
||||
|
||||
/** Drop everything. The tags on disk changed, so credits may have. */
|
||||
invalidate(): void {
|
||||
this.cache = new LRUMap<string, CachedParts>(CREDIT_CACHE_LIMIT);
|
||||
this.notify();
|
||||
}
|
||||
|
||||
private notify(): void {
|
||||
for (const fn of this.listeners) fn();
|
||||
}
|
||||
}
|
||||
|
||||
export const creditStore = new CreditStore();
|
||||
@@ -294,3 +294,81 @@ async function openAlbum(
|
||||
|
||||
navigate(target, detail);
|
||||
}
|
||||
|
||||
/**
|
||||
* One credited artist within a multi-artist credit.
|
||||
*
|
||||
* Mirrors `artist_credit_part` / `file_artists`: the name **as
|
||||
* credited** (which is not the artist's own name — MusicBrainz credits
|
||||
* "Snoop Dogg" on a track by the artist called "Snoop Doggy Dogg"), the
|
||||
* MBID to navigate to, and the literal connector that follows this
|
||||
* part.
|
||||
*/
|
||||
export interface CreditPart {
|
||||
/** The name as credited. Display uses this. */
|
||||
creditedName: string;
|
||||
/** The artist's MusicBrainz ID. Navigation uses this. */
|
||||
artistMbid: string;
|
||||
/** The connector following this part: " feat. ", " & ", ", ", "". */
|
||||
joinPhrase: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Render a credit as links, one per credited artist, with the join
|
||||
* phrases as plain text between them.
|
||||
*
|
||||
* Join phrases are **assembly instructions, not disassembly
|
||||
* instructions**. This concatenates parts; it never searches for a
|
||||
* name inside a credit string. That distinction is the whole point:
|
||||
* the stored credit text may have come from a file's tags while the
|
||||
* parts come from the catalog, and measured on a real library those
|
||||
* disagree for about one in three multi-artist credits ("Skrillex
|
||||
* feat. Swae Lee" tagged against "Skrillex & Swae Lee" upstream). A
|
||||
* search would miss, or match the wrong span. Building from parts,
|
||||
* the link boundaries are known by construction.
|
||||
*
|
||||
* Falls back to `artistLink(fallbackName, fallbackMbid)` — today's
|
||||
* behaviour exactly — when there are no parts. That is the common
|
||||
* case and not a degraded one: a single-artist credit *is* one link,
|
||||
* and a file with no recording MBID or no catalog row has nothing to
|
||||
* decompose. Do not try to split the fallback string; there is
|
||||
* genuinely no information in it to split on.
|
||||
*
|
||||
* @param parts - The credit's parts in position order, if known.
|
||||
* @param fallbackName - The credit as a single string.
|
||||
* @param fallbackMbid - The primary artist's MBID.
|
||||
*/
|
||||
export function creditLink(
|
||||
parts: readonly CreditPart[] | undefined,
|
||||
fallbackName: string,
|
||||
fallbackMbid: string,
|
||||
): TemplateResult | string {
|
||||
// One part is one link, so it is the fallback rather than a special
|
||||
// case — and a zero-part credit reaching here would otherwise
|
||||
// render as nothing at all, which is worse than the single-artist
|
||||
// answer it replaced.
|
||||
if (!parts || parts.length < 2) {
|
||||
return artistLink(fallbackName, fallbackMbid);
|
||||
}
|
||||
|
||||
return html`${parts.map(
|
||||
(part) =>
|
||||
html`${artistLink(part.creditedName, part.artistMbid)}${part.joinPhrase}`,
|
||||
)}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* The plain-text form of a credit, for `title=` attributes and any
|
||||
* other place that needs a string rather than a template.
|
||||
*
|
||||
* Rendered from the same parts by the same concatenation, so the
|
||||
* tooltip cannot disagree with the links beneath it.
|
||||
*/
|
||||
export function creditText(
|
||||
parts: readonly CreditPart[] | undefined,
|
||||
fallbackName: string,
|
||||
): string {
|
||||
if (!parts || parts.length < 2) return fallbackName;
|
||||
|
||||
return parts.map((p) => p.creditedName + p.joinPhrase).join('');
|
||||
}
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
/**
|
||||
* A track credited to more than one artist has one navigable artist in
|
||||
* this app and the rest are punctuation. `creditLink` is the fix: it
|
||||
* renders a credit as one link per credited artist with the join
|
||||
* phrases as plain text between them.
|
||||
*
|
||||
* The rule these tests exist to pin is that join phrases are
|
||||
* **assembly** instructions, not disassembly instructions — the credit
|
||||
* is built from its parts, never found by searching a name inside a
|
||||
* credit string. Measured on a real library, the stored credit text and
|
||||
* the catalog's parts disagree for about one in three multi-artist
|
||||
* credits, so a search would miss or match the wrong span.
|
||||
*/
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { html, render } from 'lit';
|
||||
|
||||
import { creditLink, creditText, type CreditPart } from '@utils/explore-link';
|
||||
|
||||
const TUPAC = '11111111-1111-4111-8111-111111111111';
|
||||
const SNOOP = '22222222-2222-4222-8222-222222222222';
|
||||
|
||||
const parts: CreditPart[] = [
|
||||
{ creditedName: '2Pac', artistMbid: TUPAC, joinPhrase: ' feat. ' },
|
||||
{ creditedName: 'Snoop Dogg', artistMbid: SNOOP, joinPhrase: '' },
|
||||
];
|
||||
|
||||
function renderToEl(value: unknown): HTMLElement {
|
||||
const host = document.createElement('div');
|
||||
render(html`${value}`, host);
|
||||
|
||||
return host;
|
||||
}
|
||||
|
||||
describe('creditLink', () => {
|
||||
it('renders one link per credited artist', () => {
|
||||
const el = renderToEl(creditLink(parts, '2Pac feat. Snoop Dogg', TUPAC));
|
||||
const links = el.querySelectorAll('a.explore-link');
|
||||
|
||||
expect(links).toHaveLength(2);
|
||||
expect(links[0]?.textContent).toBe('2Pac');
|
||||
expect(links[1]?.textContent).toBe('Snoop Dogg');
|
||||
});
|
||||
|
||||
it('puts the join phrase between the links as plain text', () => {
|
||||
const el = renderToEl(creditLink(parts, '2Pac feat. Snoop Dogg', TUPAC));
|
||||
|
||||
// The whole credit reads correctly...
|
||||
expect(el.textContent?.replace(/\s+/g, ' ').trim()).toBe(
|
||||
'2Pac feat. Snoop Dogg',
|
||||
);
|
||||
|
||||
// ...and " feat. " is not inside either link, which is the
|
||||
// difference between a credit and a link with punctuation in it.
|
||||
for (const link of el.querySelectorAll('a.explore-link')) {
|
||||
expect(link.textContent).not.toMatch(/feat/);
|
||||
}
|
||||
});
|
||||
|
||||
it('falls back to a single link when there are no parts', () => {
|
||||
const el = renderToEl(creditLink(undefined, 'Alina Baraz & Galimatias', TUPAC));
|
||||
const links = el.querySelectorAll('a.explore-link');
|
||||
|
||||
expect(links).toHaveLength(1);
|
||||
expect(links[0]?.textContent).toBe('Alina Baraz & Galimatias');
|
||||
});
|
||||
|
||||
it('does not split the fallback string on its separators', () => {
|
||||
// "&" and "with" appear inside real artist names — "Simon &
|
||||
// Garfunkel" is one artist — so a credit with no parts is one
|
||||
// link, always. This is the whole reason primaryArtist() does
|
||||
// not split on them either.
|
||||
const el = renderToEl(creditLink(undefined, 'Simon & Garfunkel', TUPAC));
|
||||
|
||||
expect(el.querySelectorAll('a.explore-link')).toHaveLength(1);
|
||||
});
|
||||
|
||||
it('treats a one-part credit as the single-link case', () => {
|
||||
// A zero- or one-part credit reaching the multi-artist branch
|
||||
// would render as nothing, or as a link with a dangling join
|
||||
// phrase after it.
|
||||
const one: CreditPart[] = [
|
||||
{ creditedName: 'Solo', artistMbid: TUPAC, joinPhrase: '' },
|
||||
];
|
||||
const el = renderToEl(creditLink(one, 'Solo', TUPAC));
|
||||
|
||||
expect(el.querySelectorAll('a.explore-link')).toHaveLength(1);
|
||||
expect(el.textContent?.trim()).toBe('Solo');
|
||||
});
|
||||
|
||||
it('renders the credited name, not the artist name', () => {
|
||||
// MusicBrainz credits "Snoop Dogg" on a track by the artist
|
||||
// called "Snoop Doggy Dogg". Display follows the credit;
|
||||
// navigation follows the MBID.
|
||||
const el = renderToEl(creditLink(parts, 'anything', TUPAC));
|
||||
|
||||
expect(el.textContent).toContain('Snoop Dogg');
|
||||
expect(el.textContent).not.toContain('Snoop Doggy Dogg');
|
||||
});
|
||||
});
|
||||
|
||||
describe('creditText', () => {
|
||||
it('reassembles the credit as a string', () => {
|
||||
expect(creditText(parts, 'ignored')).toBe('2Pac feat. Snoop Dogg');
|
||||
});
|
||||
|
||||
it('is the fallback string when there are no parts', () => {
|
||||
expect(creditText(undefined, 'Alina Baraz & Galimatias')).toBe(
|
||||
'Alina Baraz & Galimatias',
|
||||
);
|
||||
});
|
||||
|
||||
it('agrees with what creditLink renders', () => {
|
||||
// The tooltip and the links come from the same parts by the same
|
||||
// concatenation, so they cannot disagree.
|
||||
const el = renderToEl(creditLink(parts, 'ignored', TUPAC));
|
||||
|
||||
expect(el.textContent?.replace(/\s+/g, ' ').trim()).toBe(
|
||||
creditText(parts, 'ignored'),
|
||||
);
|
||||
});
|
||||
});
|
||||
+5
-1
@@ -26,7 +26,11 @@ pre-commit:
|
||||
go generate ./...
|
||||
if [ -n "$(git diff --name-only)" ]; then
|
||||
echo "Generated code is out of date. Run 'make generate' and stage the changes."
|
||||
git diff --stat
|
||||
# --no-pager, or this blocks forever on `less` waiting for a
|
||||
# keypress that a hook run without a tty will never get: the
|
||||
# commit hangs at exactly the moment it is trying to tell you
|
||||
# why it failed.
|
||||
git --no-pager diff --stat
|
||||
exit 1
|
||||
fi
|
||||
|
||||
|
||||
Reference in New Issue
Block a user