Compare commits
16
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
369810e06b | ||
|
|
e51cb13662 | ||
|
|
3d65da0529 | ||
|
|
52cbef27c4 | ||
|
|
c03c0b8ec4 | ||
|
|
8c48105ca3 | ||
|
|
1c4d6ca9a1 | ||
|
|
6bf832a4ba | ||
|
|
4f8257ef72 | ||
|
|
b505959934 | ||
|
|
409bfd5e89 | ||
|
|
0eeef6048e | ||
|
|
4fc0cdeab7 | ||
|
|
eb059a3d71 | ||
|
|
dcabec8b1d | ||
|
|
b3737d30af |
+15
-1
@@ -9,9 +9,23 @@ name: CI
|
||||
# before being written here, so every step below is a transcription of
|
||||
# something observed working rather than something expected to.
|
||||
|
||||
# **A branch push and its PR are the same commit, and testing it twice
|
||||
# costs the only runner there is.** `branches: ['**']` here meant every
|
||||
# PR booked four runs — `check` and `e2e` for the branch push, then both
|
||||
# again for `refs/pull/N/head` — on a host with capacity 1, where the
|
||||
# queue is shared with an index build that can hold it for three hours.
|
||||
#
|
||||
# `pull_request` covers feature branches, and `main` is kept because a
|
||||
# post-merge run is the record of the trunk's health. Since main now
|
||||
# refuses direct pushes, that run happens exactly once per merge.
|
||||
#
|
||||
# The trade is explicit: a branch pushed with **no** PR open gets no CI.
|
||||
# That is consistent with the workflow this repo committed to — every
|
||||
# change goes through a PR — and the signal returns the moment one is
|
||||
# opened, on the same commit.
|
||||
on:
|
||||
push:
|
||||
branches: ['**']
|
||||
branches: [main]
|
||||
pull_request:
|
||||
workflow_dispatch:
|
||||
|
||||
|
||||
@@ -7,11 +7,29 @@ name: Search index maintenance
|
||||
# import older than 6mo -> rebuild (re-import from the newest dump)
|
||||
# otherwise -> refresh (fold in new incremental listens)
|
||||
#
|
||||
# A refresh is cheap and no-ops when nothing new has been published, so
|
||||
# running it on every push to main is safe.
|
||||
# **There is deliberately no `push` trigger, and restoring one is a
|
||||
# decision rather than a cleanup.** A refresh is individually cheap, so
|
||||
# running it on every push to main looked free; what it actually does is
|
||||
# put an unattended job that mutates the only copy of a ~205 GB catalog
|
||||
# on the same trigger as an ordinary code change, on a runner with
|
||||
# capacity 1.
|
||||
#
|
||||
# That is not hypothetical. On 2026-08-17 `fix(database): retire a table
|
||||
# whose shape the schema moved past` landed on main, green — the CI
|
||||
# database is deliberately in the older encoding, so the stale-shape
|
||||
# repair judged its `explore_index` stale and dropped it, and this job
|
||||
# fell back to a full import from the dumps. `fix(database): never
|
||||
# retire the catalog the index build derives` stops that specific repair
|
||||
# and cannot undo it. Every push to main then booked another `budget`
|
||||
# (3h) of the one runner while ordinary CI queued behind it.
|
||||
#
|
||||
# So the rule this file is an instance of: **a job that mutates state
|
||||
# which cannot be rebuilt in ten minutes is triggered deliberately, not
|
||||
# by a push.** The weekly cron keeps the catalog current, and
|
||||
# workflow_dispatch resumes or forces a build — indexbuild picks up from
|
||||
# its checkpoint either way, so nothing is lost by not running on every
|
||||
# merge. See docs/index-cache.md for the snapshot and the restore.
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
schedule:
|
||||
# Weekly update pass. The 6-month rebuild is triggered by the same
|
||||
# command when it notices the import has aged out.
|
||||
@@ -33,6 +51,10 @@ on:
|
||||
|
||||
# Runs share one persistent working directory, so they must not overlap.
|
||||
# A push landing mid-build waits rather than corrupting the checkpoint.
|
||||
#
|
||||
# That directory holds the only copy of a catalog nothing can cheaply
|
||||
# re-derive: see docs/index-cache.md for the snapshot it takes and the
|
||||
# restore, which is minutes against the hours a rebuild costs.
|
||||
concurrency:
|
||||
group: search-index
|
||||
cancel-in-progress: false
|
||||
|
||||
@@ -3338,3 +3338,110 @@ Four things measured or corrected in the doing:
|
||||
- **The gate belongs before the first status write.** Declining is a
|
||||
no-op — no job in the indicator, no error tier to dismiss — which is
|
||||
what makes the refusal safe to have on by default.
|
||||
|
||||
## The stale-shape repair dropped the CI catalog (2026-08-17)
|
||||
|
||||
Not our change, but it is the operational state everything else now runs
|
||||
in, and the restore condition needs to be written down somewhere that is
|
||||
not a commit message.
|
||||
|
||||
`fix(database): retire a table whose shape the schema moved past` added
|
||||
`staleshape.go`: before `applySchema`, drop any non-Authored table whose
|
||||
live shape disagrees with the schema. That is the right rule for an
|
||||
install — a client's catalog is *downloaded*, so a stale one costs a
|
||||
minute of re-fetching the artifact, and keeping it costs every Explore
|
||||
read.
|
||||
|
||||
It runs inside `database.NewDB`, which `cmd/indexbuild` also calls. On
|
||||
the first run after it landed, 19 seconds in:
|
||||
|
||||
```
|
||||
16:15:51 retiring a table ... table=explore_index
|
||||
reason="column entity_type is TEXT, schema declares INTEGER"
|
||||
16:16:05 index maintenance mode=build reason="no completed import yet"
|
||||
lastImported=never baselineSeries=0
|
||||
```
|
||||
|
||||
**The premise was false for the one database where it was expensive.**
|
||||
That catalog is not stale; it is deliberately kept in the older text
|
||||
encoding, which `artifactStoresText` and `sourceColumns` exist to
|
||||
tolerate — so it would have been judged stale and dropped on *every*
|
||||
run. And `retireLibraryTables`, in the same package, already documents
|
||||
the opposite rule for this database: drop everything the datamap does
|
||||
**not** call Cache.
|
||||
|
||||
`fix(database): never retire the catalog the index build derives` makes
|
||||
the policy a build tag (`retireStaleCache`, false under `indexbuild`),
|
||||
which is how this project already separates the index tools. It prevents
|
||||
recurrence and cannot undo the drop: that volume was the only copy.
|
||||
|
||||
**What it cost, and the shape of the cost.** A full re-import from the
|
||||
MetaBrainz dumps, resumed across runs from a checkpoint, at a rate that
|
||||
swung between 2 and 15 MB/s. The job runs on **every push to main** with
|
||||
a 3 h budget on a runner of capacity 1 — so until the import completes,
|
||||
every push books three hours and ordinary CI queues behind it. That is
|
||||
the real damage: not one lost job, but a repeating one.
|
||||
|
||||
So the `push:` trigger in `index-artifact.yml` is **commented out**
|
||||
until a run reports `complete=true`; the weekly cron and
|
||||
`workflow_dispatch` still resume the build, which is all it needs.
|
||||
Restoring those two lines is the whole revert.
|
||||
|
||||
Three things worth keeping from it:
|
||||
|
||||
- **A repair belongs where its assumptions hold.** `NewDB` is the one
|
||||
chokepoint every binary in this project shares, including the one
|
||||
whose database cannot be re-derived cheaply. Anything destructive
|
||||
there needs to ask which binary it is in — the build tag was available
|
||||
and is what the fix used.
|
||||
- **The only copy of a 205 GB derived asset is one Docker volume.**
|
||||
There is no snapshot, so the restore time is "however long
|
||||
MetaBrainz takes today". A periodic copy would turn this class of
|
||||
incident into twenty minutes.
|
||||
- **The fix's residual trade is now the thing to watch**: with Cache
|
||||
tables never retired under `indexbuild`, a future `explore_index`
|
||||
column fails that job loudly at build time instead of silently
|
||||
rebuilding. That is the right default, and it means the next schema
|
||||
change touching `explore_index` needs a deliberate plan for this one
|
||||
database rather than none.
|
||||
|
||||
## Two guards for the index cache, and what each one is worth (2026-08-17)
|
||||
|
||||
Both come out of the incident above, and they protect different halves
|
||||
of it.
|
||||
|
||||
**`TestNoCacheTableIsRetiredHere` asserts the outcome, not the
|
||||
mechanism.** The test that shipped with the fix pins one table in one
|
||||
wrong shape, which is the failure that happened; what actually cost the
|
||||
rebuild was a destructive repair added at `database.NewDB` — the
|
||||
chokepoint every binary here shares — without asking which binary it was
|
||||
in. The next one will have a different name and a different reason. So
|
||||
this puts *every* `datamap` Cache table into a shape the schema has
|
||||
moved past, opens the database the way `cmd/indexbuild` does, and
|
||||
requires all of them to still be there.
|
||||
|
||||
Three things it got right by being written this way. The table list is
|
||||
`datamap.ByKind(Cache)`, so the two credit tables added the same day
|
||||
were covered without anyone adding them — flipping the policy back fails
|
||||
on **five** tables including `artist_credit_part` and
|
||||
`artist_credit_ref`, where the single-table test fails on one. It
|
||||
asserts rows survive as well as the table, because SQLite does an
|
||||
implicit DELETE before a DROP and a repair that recreated the table
|
||||
would otherwise look identical. And it *accepts* an error from `NewDB`,
|
||||
because that is the trade the fix documents: loud failure instead of a
|
||||
silent day of downloading.
|
||||
|
||||
**`scripts/index-cache-snapshot.sh` covers the half no test can.** The
|
||||
volume held the only copy of a catalog whose rebuild is hours of someone
|
||||
else's bandwidth. `VACUUM INTO` rather than `cp`, because a byte copy of
|
||||
a live SQLite file is a corrupt file of plausible size; the staging
|
||||
directory is deliberately not copied, since a build resumes without it;
|
||||
and the snapshot is reopened and asked for its catalog row count before
|
||||
any rotation happens. Both failure paths were exercised rather than
|
||||
argued: a corrupt source and an empty catalog each exit non-zero, delete
|
||||
their own output, and leave the previous snapshots in place.
|
||||
|
||||
`docs/index-cache.md` is the restore procedure, and the number that
|
||||
makes it worth having: a restored snapshot resolves to `refresh` and
|
||||
folds in the incremental listens since — minutes, against the 3–23 h a
|
||||
rebuild was estimating.
|
||||
|
||||
@@ -0,0 +1,337 @@
|
||||
# 015 — Multi-artist credits, navigable
|
||||
|
||||
## The problem
|
||||
|
||||
A track credited to more than one artist has exactly one navigable
|
||||
artist in this app, and the others are punctuation.
|
||||
|
||||
`audio_files` carries `artist_credit` (the credit as tagged, for
|
||||
display) and `artist_id` (one artist, for grouping and browsing).
|
||||
`primaryArtist()` (`backend/library/artistcredit.go:53`) resolves that
|
||||
one artist by *string-parsing* the credit: it strips a " feat. "
|
||||
clause, and deliberately does not split on `&`, `x`, `with` or `,`
|
||||
because those appear inside real artist names. So "Lana Del Rey ft.
|
||||
Sean Lennon" stores Lana Del Rey and discards Sean Lennon entirely,
|
||||
and "Alina Baraz & Galimatias" stores one artist whose name is the
|
||||
whole credit.
|
||||
|
||||
### What the measurement says
|
||||
|
||||
Measured 2026-08-16 against a real 26,069-file library (19,840 mp3,
|
||||
6,229 flac; 57 unreadable, m4a/ogg not examined), plus an 80+80
|
||||
MusicBrainz `inc=artist-credits` sample.
|
||||
|
||||
- **13%** of a random sample of the library's recordings have more
|
||||
than one credited artist in MusicBrainz (10 of 79 resolved).
|
||||
Extrapolates to ~3,250 of the 24,989 files carrying a recording
|
||||
MBID.
|
||||
- **0.86%** of files (224) carry any structured multi-artist signal in
|
||||
their own tags. mp3 carries **zero** files with multiple
|
||||
`MUSICBRAINZ_ARTISTID` values across 19,840 files; flac has 87.
|
||||
- **1,286** files say "feat." in `ARTIST`; **1,159 of them (90%)**
|
||||
have nothing structured behind it. A sample of 80 such files was
|
||||
multi-artist in MB **80 of 80 times**.
|
||||
|
||||
CLAUDE.md currently justifies plan 013's removal of `artist_credit` /
|
||||
`artist_credit_artist` with "3 credits of 2,823 listed more than one
|
||||
artist". That figure measured **our own writer**, not the library:
|
||||
`cachedLinkArtist` was called exactly once per credit
|
||||
(`e7748f1^:backend/library/library.go:1842`), so a collaboration could
|
||||
never have been recorded, and the three were resolution collisions on
|
||||
shared credit text. Dropping the join table was still correct — it only
|
||||
ever held one row, so it was pure join cost — but the stated evidence
|
||||
does not support "multi-artist is rare". Correcting that claim is part
|
||||
of this plan.
|
||||
|
||||
### Why the tags cannot answer it
|
||||
|
||||
Deriving the decomposition locally, with no network, works **79% of the
|
||||
time** (169 of 215 files with a multi-value `ARTISTS` tag: mp3 69/105,
|
||||
flac 100/110), and the failures are systematic rather than random:
|
||||
|
||||
```
|
||||
ARTIST = '2Pac feat. Snoop Dogg, Nate Dogg, Hussein Fatal & Yaki Kadafi'
|
||||
ARTISTS = ['2Pac', 'Snoop Doggy Dogg', 'Nate Dogg', 'Fatal', 'Yaki Kadafi']
|
||||
```
|
||||
|
||||
`ARTISTS` holds **canonical** artist names; `ARTIST` holds
|
||||
**as-credited** names. Locating one inside the other fails on
|
||||
"Snoop Doggy Dogg" vs "Snoop Dogg", on "Fatal" vs "Hussein Fatal", and
|
||||
on Unicode (`Michel'le` vs `Michel’le`, `K-Ci` vs `K‐Ci` — U+2010, not
|
||||
a hyphen). That distinction is precisely what a join phrase encodes,
|
||||
and it is why this cannot be a tag-parsing feature.
|
||||
|
||||
Two format details that will mislead anyone re-running the probe:
|
||||
Picard writes `ARTISTS` **slash-joined into one TXXX frame** on mp3 and
|
||||
as **true repeated Vorbis keys** on flac, so a probe splitting only on
|
||||
NUL undercounts mp3 to zero.
|
||||
|
||||
## The shape
|
||||
|
||||
MusicBrainz models a credit as ordered parts, and the credit *string*
|
||||
is derived from them — `artist_credit.name` is a cached render, nothing
|
||||
more. Each participant is `(position, artist, name, join_phrase)`,
|
||||
where `artist` is the MBID (canonical, what you navigate to) and `name`
|
||||
is the credited spelling (what you display).
|
||||
|
||||
**Join phrases are assembly instructions, not disassembly
|
||||
instructions.** Rendering is a concatenation, never a search:
|
||||
|
||||
```
|
||||
for each (position, artist_mbid, credited_name, join_phrase):
|
||||
emit link(credited_name -> artist_mbid)
|
||||
emit text(join_phrase)
|
||||
```
|
||||
|
||||
The link positions are known **by construction**. This is load-bearing:
|
||||
if we instead located each `credited_name` inside the stored
|
||||
`artist_credit` text, we would reintroduce the mismatch above — the
|
||||
stored string may have come from the tags while the parts come from the
|
||||
catalog, and those **disagree for ~1 in 3 multi-artist files** (61 of
|
||||
90 sampled credits rendered exactly equal to the tag string).
|
||||
Divergences seen: `'Skrillex feat. Swae Lee'` tagged vs
|
||||
`'Skrillex & Swae Lee'` in MB; `'STRFKR'` vs `'Starfucker'`;
|
||||
`'Zedd feat. Hayley Williams'` vs `'... of Paramore'`. Either MB was
|
||||
edited after tagging or Picard versions differ; either way the search
|
||||
would miss or match the wrong span.
|
||||
|
||||
So `audio_files.artist_credit` stops being the source of truth and
|
||||
becomes the **fallback**, used only where there are no parts.
|
||||
|
||||
## Where the data comes from
|
||||
|
||||
The catalog carries the decomposition; no user ever makes a
|
||||
per-recording call. Two sources were ruled out first, both cheaply:
|
||||
|
||||
- **The canonical dump — which is what CI already pulls
|
||||
(`dumpimport.go:84-85`) — does not have it.**
|
||||
`canonical_musicbrainz_data.csv` gives `artist_mbids` (ordered list)
|
||||
and `artist_credit_name`, but that last column is the *rendered*
|
||||
string. Splitting it on CI needs the as-credited names, so CI would
|
||||
fail exactly the way a local parse does.
|
||||
- **The JSON dumps do not cover the catalog.**
|
||||
`json-dumps/recording.tar.xz` is 31 MB / 368 MB uncompressed and
|
||||
holds **153,691 recordings**, not ~35M. Measured against the test
|
||||
library's 24,885 recording MBIDs: **0.00% overlap, zero rows**. It is
|
||||
some other subset and is not usable.
|
||||
|
||||
That leaves the core dump, **`mbdump.tar.bz2`** (7.1 GB compressed at
|
||||
the 20260815 export), from
|
||||
`https://data.metabrainz.org/pub/musicbrainz/data/fullexport/`. Four
|
||||
members are needed:
|
||||
|
||||
| member | why | approx rows |
|
||||
| --- | --- | --- |
|
||||
| `mbdump/artist_credit_name` | `(artist_credit, position, artist, name, join_phrase)` — the payload | ~4M |
|
||||
| `mbdump/artist` | `id -> gid`, since the above references artist *row ids* | ~2.6M |
|
||||
| `mbdump/recording` | `gid -> artist_credit`, to key credits by recording MBID | ~35M |
|
||||
| `mbdump/release_group` | same, for album credits | ~2M |
|
||||
|
||||
### Coverage is not a concern
|
||||
|
||||
Of 24,885 distinct recording MBIDs in the test library, **24,808
|
||||
(99.7%)** already have an `explore_index` recording row, measured
|
||||
against a database at 2,052,200 rows — i.e. shipped-artifact coverage,
|
||||
not a local build's. The popularity filter does not strand the long
|
||||
tail here.
|
||||
|
||||
## Status
|
||||
|
||||
- **Phase 1 — done.** `backend/explore/dumpcredits.go` +
|
||||
`dumpcreditswrite.go`, wired into `dumpimport.go`'s `run` behind its
|
||||
own `credits_import_done` marker.
|
||||
- **Phase 2 — done.** `cmd/indexexport` writes the two tables;
|
||||
`artifactimport.go` reads them behind `artifactHasCredits()`.
|
||||
- **Phase 4 — done, and it does not need Phase 3.** `explore.GetCredits`
|
||||
reads the catalog tables keyed on the *recording* MBID, which both
|
||||
sides of the app already carry — a catalog row has one and so does a
|
||||
local file (`library.Track.RecordingMBID`). So one binding serves the
|
||||
Explore pages and the library's own lists, and all ten artist-link
|
||||
call sites render credits today without a local table.
|
||||
- **Phase 3 (`file_artists`) — not started, and now an
|
||||
offline-resilience task rather than a prerequisite.** The table is
|
||||
deliberately *not* declared yet: nothing writes or reads it, and a
|
||||
schema file plus a datamap note describing behaviour that does not
|
||||
exist is a claim the code cannot back. Its remaining
|
||||
value is that credits currently vanish when the catalog is absent or
|
||||
still downloading, which is precisely the `no-index` state
|
||||
`ShelfPage.State` exists to describe. Materialising into
|
||||
`file_artists` is what makes a library stand on its own.
|
||||
|
||||
**Nothing renders yet in practice**, because no published artifact
|
||||
carries credit tables — every credit falls back to its single link
|
||||
until an index build with Phase 1 runs and is exported.
|
||||
|
||||
**Column layouts are verified against the real 20260815 export**, not
|
||||
taken from the schema docs — `artist(id, gid, …)`,
|
||||
`artist_credit(id, name, artist_count, …)`,
|
||||
`artist_credit_name(credit, position, artist, name, join_phrase)` and
|
||||
`recording(id, gid, name, artist_credit, …)` were each read out of the
|
||||
dump. `release_group` shares `recording`'s first four columns and is
|
||||
the one layout still taken on trust; `ErrDumpShape` turns a wrong guess
|
||||
into a loud failure rather than a quietly wrong catalog.
|
||||
|
||||
**Still unrun: the ingest against the real 7.1 GB dump.** Everything is
|
||||
covered by tests over a synthetic tar, which cannot catch a surprise in
|
||||
the other ~35M rows.
|
||||
|
||||
### Phase 1 — Ingest credits on CI
|
||||
|
||||
New dump stage in `cmd/indexbuild`, behind the `indexbuild` tag with
|
||||
the rest of `dumpimport.go`'s stages.
|
||||
|
||||
**Constraint from `b98840e`:** `cmd/indexbuild` is built
|
||||
`CGO_ENABLED=0` in a plain `golang` container and must not reach the
|
||||
Wails `application` package — `TestIndexToolsDoNotImportWails` walks
|
||||
`go list -deps -tags indexbuild`. Nothing here should need it, but a
|
||||
new `ServiceStartup` hook on a package this imports is how it comes
|
||||
back. Go's `compress/bzip2` is pure Go and decompress-only, which is
|
||||
all this needs.
|
||||
|
||||
**Measured, 20260815 export.** Tar members are **alphabetical**, and
|
||||
that is favourable: `artist` (435 MB), `artist_credit` (414 MB) and
|
||||
`artist_credit_name` (237 MB) all fall inside the first ~900 MB
|
||||
compressed, while `recording` and `release_group` come later. So the
|
||||
maps are complete before the rows that consume them arrive, and no
|
||||
recording data is ever buffered.
|
||||
|
||||
Pure-Go `compress/bzip2` decompresses at **26 MB/s uncompressed /
|
||||
8.7 MB/s compressed** (measured on a 250 MB prefix, 3.01x ratio) —
|
||||
**~13.7 min** for the whole file single-threaded, and less because the
|
||||
stream can stop after `release_group` rather than reading the
|
||||
`series`/`tag`/`track`/`url`/`work` tail. The 2 MB/s origin throttle
|
||||
dominates, as it already does for every other dump here.
|
||||
|
||||
Do not, however, *depend* on the ordering: assert it and fall back to
|
||||
buffering if a future export reorders, rather than silently emitting
|
||||
nothing.
|
||||
|
||||
- `artist` -> `map[int32]uuid16` (~2.6M x ~20 B = ~60 MB)
|
||||
- `artist_credit_name` -> `map[int32][]creditPart` (~4M x ~40 B =
|
||||
~200 MB)
|
||||
- `recording` / `release_group` -> emit `gid -> credit_id` **only for
|
||||
MBIDs already in `explore_index`** (the kept set is ~1.4M x 16 B =
|
||||
~22 MB), which is what keeps 35M rows from being held
|
||||
|
||||
Peak ~300 MB, one sequential pass.
|
||||
|
||||
**Only multi-artist credits are stored.** A single-artist credit is
|
||||
`(name, "")` and is already fully described by `explore_index`'s
|
||||
`artist_name` / `artist_mbid`; storing it would triple the table for
|
||||
nothing. Post-filter after loading, once the row count per credit is
|
||||
known.
|
||||
|
||||
New tables (and `datamap` entries, or `TestCatalogCoversSchema` fails
|
||||
the build — both are `Cache`, matching `explore_index`):
|
||||
|
||||
```
|
||||
artist_credit_part(credit_id, position, artist_mbid, credited_name, join_phrase)
|
||||
```
|
||||
|
||||
with `explore_index.artist_credit_id` as the link. Credits are
|
||||
**shared** — an album's twelve tracks by one artist share one credit
|
||||
row — which is the opposite of 013's local verdict, and correctly so:
|
||||
1:1 in a local library, genuinely many-to-one at 2M-row catalog scale.
|
||||
|
||||
### Phase 2 — Ship them in the artifact
|
||||
|
||||
`cmd/indexexport` currently creates exactly two tables in the artifact
|
||||
(`explore_index`, `artifact_meta`, at `cmd/indexexport/*.go:147,170`),
|
||||
so this is a structural addition, not a column.
|
||||
|
||||
Estimated size: ~13% of 1.4M recordings, deduplicated by shared credit,
|
||||
at ~2.3 parts each — order 400k rows, ~18 MB uncompressed. Against a
|
||||
~0.6 GB install that is acceptable; it must be measured rather than
|
||||
assumed before merge.
|
||||
|
||||
`artifactimport.go` must read it **only if present**, on the writer
|
||||
handle where `core` is attached — the `artifactHasTotals()` /
|
||||
`artifactStoresText()` pattern (`artifactimport.go:145-175`), one step
|
||||
up from a column to a table. An artifact published before this exists
|
||||
is still a perfectly good catalog and must import as one that declines
|
||||
to answer. Adding this to the importer's SELECT list without the probe
|
||||
is how every already-published artifact starts failing.
|
||||
|
||||
`artifactCatalogColumns` gains `artist_credit_id`; it is kept in sync
|
||||
with the exporter by `TestArtifactColumnsMatchExporter`.
|
||||
|
||||
### Phase 3 — Materialize locally
|
||||
|
||||
```
|
||||
file_artists(audio_file_id, position, artist_id, credited_name, join_phrase)
|
||||
```
|
||||
|
||||
`credited_name` is stored **per row**, not looked up from
|
||||
`artists.name` — that is the Snoop-Doggy-Dogg distinction, and it is
|
||||
the whole point.
|
||||
|
||||
Filled at scan/import time by joining `audio_files.recording_mbid`
|
||||
against the catalog. **Materialized rather than resolved live**,
|
||||
because the catalog is a downloaded artifact that can be absent or
|
||||
still arriving — that is why `ShelfPage.State` has a `no-index` value —
|
||||
and a library whose track rows lose their artists when the catalog is
|
||||
missing is worse than today.
|
||||
|
||||
That implies a backfill for the case where the catalog arrives *after*
|
||||
the library was scanned. It registers with `jobs` (progress, cancel)
|
||||
like every other long pass, and takes a **distinct kind** from
|
||||
`index-build`, since `job-controls.ts` keys its "you will discard hours
|
||||
of downloading" confirmation on that kind.
|
||||
|
||||
`artists` gains rows for guests who own no files. **This changes what
|
||||
the artists grid shows** and is an open question below.
|
||||
|
||||
### Phase 4 — Render
|
||||
|
||||
`utils/explore-link.ts` gains a credit-rendering entry point taking
|
||||
ordered parts and returning a `TemplateResult`. Every row and detail
|
||||
view already renders artist names through it, so they inherit
|
||||
multi-artist links without individually knowing credits exist — the
|
||||
property that made centralising it worthwhile.
|
||||
|
||||
Its existing fallback philosophy already covers the no-parts case: "a
|
||||
list where some rows are clickable and others silently are not reads as
|
||||
a bug, not as a statement about metadata." Where there are no parts
|
||||
(no recording MBID, or no catalog row — ~4% of the test library) render
|
||||
today's behaviour: the flat `artist_credit` string with one link to the
|
||||
primary artist. **Do not split the string there.** There is genuinely
|
||||
no information to split on, and that is the one place the temptation
|
||||
returns.
|
||||
|
||||
`primaryArtist()` stays exactly as it is. It remains the fallback and
|
||||
is still what `artist_id` means.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **Catalog credit vs tagged credit, when they disagree** (~1 in 3
|
||||
multi-artist files). Rendering the catalog's decomposition is what
|
||||
makes names navigable; preserving the file's is what makes the app
|
||||
reflect the user's files. Leaning toward: render the catalog
|
||||
decomposition, keep `artist_credit` as the fallback string. Wants a
|
||||
deliberate decision, not an accident.
|
||||
2. **Do guest artists appear in the artists grid?** Phase 3 creates
|
||||
`artists` rows for people who own no files. The grid currently means
|
||||
"artists in your library" and joins `audio_files`. A guest on one
|
||||
track is arguably in the library and arguably not. Whichever way,
|
||||
the ownership question stays "is there a file" — that rule does not
|
||||
bend.
|
||||
3. **`release_group` credits** are ingested in the same pass for
|
||||
nearly nothing, but album-artist rendering is a separate surface.
|
||||
Ship the data in phase 1, render in a follow-up rather than widening
|
||||
phase 4.
|
||||
4. **Our own `tagwriter`** does not write `ARTISTS` or multiple
|
||||
`MUSICBRAINZ_ARTISTID` frames, so autotagging a folder degrades the
|
||||
very field this rests on — the same shape as the existing
|
||||
track-totals note. Out of scope here; worth recording.
|
||||
|
||||
## Verification
|
||||
|
||||
- Coverage: re-run the library probe and assert `file_artists` is
|
||||
populated for ~13% of files, not ~0.9%.
|
||||
- `TestCatalogCoversSchema` / `TestLifetimesMatchSchema` for the new
|
||||
tables.
|
||||
- `TestIndexToolsDoNotImportWails` still passes with the new stage.
|
||||
- An artifact **without** the credits table imports cleanly (the
|
||||
`artifactHasTotals` regression shape).
|
||||
- Round-trip: a known multi-artist recording renders each name as a
|
||||
separate link with the correct join phrases between them.
|
||||
@@ -233,6 +233,74 @@ rather than renaming them.
|
||||
the drift it caused before — `sql/schemas/` and the migrations
|
||||
disagreed, and sqlc generated against the stale one.
|
||||
|
||||
**What that costs an existing database is repaired once, at open.**
|
||||
`CREATE ... IF NOT EXISTS` reaches an existing table only if its shape
|
||||
already matches and otherwise silently no-ops, so a *changed* table
|
||||
never migrates. Plan 014 added `total_tracks` to `explore_index` and
|
||||
to `indexRowFields` — the projection every explore read uses — and no
|
||||
database that already existed grew the column: **every** Explore
|
||||
search, browse, artist and album page on such an install failed with
|
||||
`no such column: total_tracks`, while a fresh install was perfectly
|
||||
healthy, which is exactly why no test saw it. Plan 013 was worse on
|
||||
the same install: `applySchema` could not be applied at all over a
|
||||
pre-013 `audio_files`, so the app did not open.
|
||||
|
||||
`backend/database/staleshape.go` runs before `applySchema` and
|
||||
retires what is stale, so the create is a create. Five things about
|
||||
it are load-bearing:
|
||||
- **It parses `sql/schemas/` for the expectation** rather than
|
||||
writing the column list down a second time, because a second list
|
||||
is a second thing to forget — the fault it exists to repair.
|
||||
- **It notices a changed *type*, not just a missing column.** 013
|
||||
moved `mbid` from TEXT to BLOB, and SQLite does not coerce between
|
||||
them: a comparison against 16 raw bytes returns no rows rather than
|
||||
an error. `ALTER TABLE ADD COLUMN` would have handled
|
||||
`total_tracks` alone and cannot express this at all, which is why
|
||||
the repair drops rather than migrates.
|
||||
- **`Authored` is never retired**, and that boundary is a test
|
||||
(`TestAuthoredTablesAreNeverRetired`), not a comment. Everything
|
||||
else is rebuildable: `Cache` by definition, `Owned` by a rescan —
|
||||
plan 013's stated "delete and rescan" — and `Derived` from Owned.
|
||||
A table the schema no longer describes at all goes too; 013 left
|
||||
seven behind plus `schema_migrations`.
|
||||
- **Whether a stale `Cache` table may be rebuilt is a build tag**, and
|
||||
it is the most expensive thing in this file to get wrong. In the app
|
||||
the catalog is *downloaded*, so a wrong shape costs a minute of
|
||||
re-fetching the artifact and keeping it costs every Explore read. In
|
||||
`cmd/indexbuild` the catalog is *derived*, and the only way back is
|
||||
the ~205 GB dump stream the `/cache` volume exists to avoid — so
|
||||
`retireStaleCache` is false there (`staleshape_policy_indexbuild.go`)
|
||||
and `TestTheCatalogSurvivesAStaleShape` fails the moment it is not.
|
||||
`TestNoCacheTableIsRetiredHere` is the same assertion made of *every*
|
||||
`datamap` Cache table rather than one, because the risk is not that
|
||||
shape recurring — it is the next destructive repair added to
|
||||
`database.NewDB`, the chokepoint every binary here shares, without
|
||||
asking which binary it is in.
|
||||
This is written down because it already happened: the repair shipped
|
||||
without the distinction and dropped the real CI catalog on its first
|
||||
run, with `reason="column entity_type is TEXT, schema declares
|
||||
INTEGER"`. The mismatch was genuine — that database is deliberately
|
||||
kept in the older encoding, which `fix(indexexport): read an index
|
||||
older than the binary` exists to tolerate — so it would have been
|
||||
dropped on *every* run. The consequence is that a future
|
||||
`explore_index` column fails the index job loudly on `applySchema`
|
||||
rather than silently costing it a rebuild, which is the trade a
|
||||
human should get to make.
|
||||
- **The drops are one transaction with `defer_foreign_keys`.** Those
|
||||
legacy tables reference each other, so dropping them in any order
|
||||
fails on whichever goes first, and turning foreign keys *off*
|
||||
instead would silently take `playlist_tracks.audio_file_id`'s
|
||||
ON DELETE SET NULL with it — leaving playlist entries pointing at
|
||||
ids a rescan reissues to *different songs*. Nulled entries are
|
||||
empty; stale ones are wrong, and wrong quietly.
|
||||
- **The order is sorted, so a failure reproduces.** Map order is
|
||||
random, and the foreign-key bug above passed its own regression
|
||||
test on two runs in three until the order was fixed.
|
||||
|
||||
Retiring `explore_index` takes its FTS and its meta with it, because
|
||||
the `dump_import_done` marker is what would otherwise stop the
|
||||
artifact ever being fetched again.
|
||||
|
||||
**What that costs an existing database is that it does not open**, and
|
||||
"delete and rescan" is the answer (plan 013, open question 1) — free
|
||||
for everyone except one machine. The index job's `/cache` volume is a
|
||||
@@ -508,6 +576,56 @@ selected as a literal `0`. Adding the column to the importer's SELECT
|
||||
list without that is how a published artifact — which nobody can re-cut
|
||||
retroactively — starts failing with `no such column`.
|
||||
|
||||
**A credit is ordered parts, and the string is derived from them.** A
|
||||
track credited to several artists had exactly one navigable artist and
|
||||
the rest were punctuation: `primaryArtist()` string-parses the credit,
|
||||
strips a " feat. " clause and discards the guest, and deliberately does
|
||||
not split on `&`, `with` or `,` because those live inside real artist
|
||||
names ("Simon & Garfunkel"). Measured on a real 26,069-file library,
|
||||
**13%** of recordings are multi-artist upstream while only **0.86%** of
|
||||
files carry a structured multi-artist tag — mp3 carries *zero* files
|
||||
with multiple `MUSICBRAINZ_ARTISTID` across 19,840 — so this cannot be
|
||||
a tag-parsing feature. (The "3 credits of 2,823" figure that justified
|
||||
plan 013's removal of the credit tables measured our own *writer*:
|
||||
`cachedLinkArtist` ran once per credit, so a collaboration could never
|
||||
have been recorded. Dropping the join table was still right on cost.)
|
||||
|
||||
`artist_credit_part` / `artist_credit_ref` carry the decomposition for
|
||||
multi-artist credits only — a single-artist credit is already
|
||||
`explore_index`'s own `artist_name`, and storing those would triple the
|
||||
table to say nothing. Five things about it are load-bearing:
|
||||
|
||||
- **Join phrases are assembly instructions, not disassembly ones.**
|
||||
`creditLink` concatenates parts, so link boundaries are known by
|
||||
construction. Locating a `credited_name` *inside* the stored credit
|
||||
string would reintroduce the fault this exists to fix: that string may
|
||||
come from the file's tags while the parts come from the catalog, and
|
||||
the two disagree for ~1 in 3 multi-artist credits (`'Skrillex feat.
|
||||
Swae Lee'` tagged against `'Skrillex & Swae Lee'` upstream).
|
||||
- **`credited_name` is stored per row**, never joined from `artists`:
|
||||
MusicBrainz credits "Snoop Dogg" on a track by the artist called
|
||||
"Snoop Doggy Dogg". Display follows the credit, navigation the MBID.
|
||||
- **The lookup is keyed on the recording MBID**, which the catalog and a
|
||||
local file both carry (`library.Track.RecordingMBID`), so one binding
|
||||
serves Explore and the library's own lists — which is why this needed
|
||||
no local table. `file_artists` remains the offline-resilience step and
|
||||
is deliberately *not* declared until something writes it.
|
||||
- **Absence is cached as an answer.** `credit-store.ts` stores `[]` for
|
||||
a single-artist credit — *asked*, not *answered* — or the ~87% that
|
||||
have nothing to decompose are re-requested on every render forever.
|
||||
`request()` is per-row and coalesces into one call per frame, because
|
||||
a virtualized list cannot hand over "the whole list": 50,000 rows is
|
||||
100 queries for the ~30 on screen.
|
||||
- **The dump is a third source, and it had to be.** The canonical dump
|
||||
CI already streams has no join phrases and no as-credited names, and
|
||||
the JSON dumps cover 153,691 recordings of ~35M with *zero* overlap
|
||||
against a real library. So `mbdump.tar.bz2` — 7.1 GB, ~13.7 min in
|
||||
pure-Go bzip2, whose members are alphabetical, which is what lets one
|
||||
pass resolve an entity's credit without buffering 35M recordings. The
|
||||
pass runs on **every** mode, because a complete import means
|
||||
`refresh`, which never enters the importer at all, and it reports
|
||||
whether it populated anything so `changed` republishes the artifact.
|
||||
|
||||
**A 0.6 GB download asks about the connection first.** `explore`'s
|
||||
catalog artifact had no network awareness at all, which on a phone is a
|
||||
month's data allowance spent without being asked (plan 016 B4).
|
||||
|
||||
@@ -179,10 +179,12 @@ bindings-check: ## Fail if the generated bindings are stale
|
||||
css-check: ## Fail if a css`` literal was ended early by a backtick in a comment
|
||||
@cd frontend && node scripts/check-css-literals.mjs
|
||||
|
||||
# .pi/ documents commands, and a skill that documents a command wrongly
|
||||
# is worse than no skill: an agent runs it confidently. Every command
|
||||
# in there is a make target on purpose, so this is checkable.
|
||||
skill-check: ## Fail if .pi/ documents a make target that does not exist
|
||||
# .pi/ and CLAUDE.md document commands, and a doc that documents a
|
||||
# command wrongly is worse than no doc: an agent runs it confidently.
|
||||
# Every command in them is a make target on purpose, so this is
|
||||
# checkable. It also asserts AGENTS.md is a symlink to CLAUDE.md, so the
|
||||
# two harnesses cannot drift onto two descriptions of one project.
|
||||
skill-check: ## Fail if the agent docs name a missing make target, or AGENTS.md is not a symlink
|
||||
@./scripts/skill-check.sh
|
||||
|
||||
# Conventional Commits, which CLAUDE.md claimed CI enforced for a long
|
||||
|
||||
@@ -89,6 +89,14 @@ func NewDB(logger *slog.Logger) (*DB, error) {
|
||||
return nil, fmt.Errorf("could not apply PRAGMAs: %w", err)
|
||||
}
|
||||
|
||||
// Before the schema is applied, not after: applySchema is
|
||||
// CREATE ... IF NOT EXISTS, which no-ops against a table that
|
||||
// already exists in an older shape. Retiring the stale one first is
|
||||
// what turns that no-op into a create.
|
||||
if err := retireStaleTables(dbCtx, db, logger); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if err := applySchema(dbCtx, db); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
-- The decomposition of a multi-artist credit, from the MusicBrainz
|
||||
-- dump. One row per credited artist, in credit order.
|
||||
--
|
||||
-- A credit is ordered parts, and the credit *string* is derived from
|
||||
-- them -- MusicBrainz's own `artist_credit.name` is a cached render and
|
||||
-- nothing more. Rendering is a concatenation:
|
||||
--
|
||||
-- for each part in position order:
|
||||
-- emit link(credited_name -> artist_mbid)
|
||||
-- emit text(join_phrase)
|
||||
--
|
||||
-- so the link boundaries are known by construction. That is the whole
|
||||
-- reason this table exists, and it is why nothing may reconstruct a
|
||||
-- credit by *searching* for a name inside a credit string: the stored
|
||||
-- string may have come from a file's tags while the parts come from the
|
||||
-- catalog, and measured on a real library those disagree for about one
|
||||
-- in three multi-artist credits ("Skrillex feat. Swae Lee" tagged
|
||||
-- against "Skrillex & Swae Lee" upstream). A search would miss, or
|
||||
-- match the wrong span.
|
||||
--
|
||||
-- `credited_name` is the name *as credited*, which is not the artist's
|
||||
-- canonical name: MusicBrainz credits "Snoop Dogg" on a track by the
|
||||
-- artist whose name is "Snoop Doggy Dogg". It is stored per row rather
|
||||
-- than joined from an artist table for exactly that reason.
|
||||
--
|
||||
-- Only *multi-artist* credits are stored. A single-artist credit is
|
||||
-- (name, "") and is already fully described by explore_index's
|
||||
-- artist_name and artist_mbid; storing those would roughly triple the
|
||||
-- table to say nothing new.
|
||||
--
|
||||
-- Credits are shared: an album's twelve tracks by one artist reference
|
||||
-- one credit_id. That is the opposite of the local library's verdict
|
||||
-- in plan 013, and correctly so -- credit sharing is 1:1 in one
|
||||
-- person's files and genuinely many-to-one across a 2M-row catalog.
|
||||
--
|
||||
-- MBIDs are the same 16 raw bytes explore_index stores, for the same
|
||||
-- size reason and with the same CHECK, so a stringly write fails at the
|
||||
-- insert that made it rather than reading back as no rows at all. See
|
||||
-- backend/explore/mbid.go.
|
||||
CREATE TABLE IF NOT EXISTS artist_credit_part (
|
||||
credit_id INTEGER NOT NULL,
|
||||
position INTEGER NOT NULL,
|
||||
artist_mbid BLOB NOT NULL CHECK(length(artist_mbid) = 16),
|
||||
|
||||
-- The name as credited on this release, which may differ from the
|
||||
-- artist's canonical name. Display uses this; navigation uses the
|
||||
-- MBID above.
|
||||
credited_name TEXT NOT NULL,
|
||||
|
||||
-- The literal connector that follows this part -- " feat. ", " & ",
|
||||
-- ", ", or "" on the last part. Rendered as plain text between two
|
||||
-- links.
|
||||
join_phrase TEXT NOT NULL DEFAULT '',
|
||||
|
||||
PRIMARY KEY (credit_id, position)
|
||||
) WITHOUT ROWID;
|
||||
@@ -0,0 +1,30 @@
|
||||
-- Which credit a catalog entity is credited to. One row per recording
|
||||
-- or release group whose credit names more than one artist.
|
||||
--
|
||||
-- This is a table rather than an `explore_index.artist_credit_id`
|
||||
-- column, and that is a deliberate consequence of how this app applies
|
||||
-- its schema. `applySchema` is CREATE ... IF NOT EXISTS and there is
|
||||
-- no migration chain (plan 013), so a *column* added to an existing
|
||||
-- table never reaches a database that already has it -- while a new
|
||||
-- *table* is created on every install, old or new, for free.
|
||||
-- explore_index is the one table nobody can afford to drop and rebuild
|
||||
-- on a schema change: it is the artifact users download rather than
|
||||
-- derive.
|
||||
--
|
||||
-- Only multi-artist credits are referenced here, matching
|
||||
-- artist_credit_part. An entity with no row is credited to exactly one
|
||||
-- artist, which explore_index's own artist_name and artist_mbid already
|
||||
-- describe -- so absence is the common case and means "nothing to
|
||||
-- decompose", not "unknown".
|
||||
--
|
||||
-- `credit_id` is opaque and is only meaningful against the
|
||||
-- artist_credit_part rows built or imported alongside it. The two are
|
||||
-- always written together; nothing persists a credit_id anywhere else.
|
||||
-- The local library stores resolved parts, never this id.
|
||||
CREATE TABLE IF NOT EXISTS artist_credit_ref (
|
||||
mbid BLOB NOT NULL PRIMARY KEY CHECK(length(mbid) = 16),
|
||||
credit_id INTEGER NOT NULL
|
||||
) WITHOUT ROWID;
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_artist_credit_ref_credit
|
||||
ON artist_credit_ref(credit_id);
|
||||
@@ -27,6 +27,19 @@ type Artist struct {
|
||||
Mbid sql.NullString
|
||||
}
|
||||
|
||||
type ArtistCreditPart struct {
|
||||
CreditID int64
|
||||
Position int64
|
||||
ArtistMbid []byte
|
||||
CreditedName string
|
||||
JoinPhrase string
|
||||
}
|
||||
|
||||
type ArtistCreditRef struct {
|
||||
Mbid []byte
|
||||
CreditID int64
|
||||
}
|
||||
|
||||
type ArtistEnrichment struct {
|
||||
ArtistMbid string
|
||||
BrowsedAt sql.NullTime
|
||||
|
||||
@@ -0,0 +1,560 @@
|
||||
package database
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"log/slog"
|
||||
"maps"
|
||||
"path"
|
||||
"slices"
|
||||
"strings"
|
||||
|
||||
"yellowjacket/backend/datamap"
|
||||
)
|
||||
|
||||
// This file repairs the one thing `CREATE TABLE IF NOT EXISTS` cannot.
|
||||
//
|
||||
// `sql/schemas/` is the single description of the schema and there is no
|
||||
// migration chain (plan 013): a schema change is one edit to one file.
|
||||
// That works perfectly for a *new* table, which every install then
|
||||
// creates, and not at all for a changed one -- `IF NOT EXISTS` reaches
|
||||
// an existing table only if its shape already matches, and otherwise
|
||||
// silently no-ops. The user's answer to that is "delete and rescan"
|
||||
// (plan 013, open question 1), which is free for everything a rescan
|
||||
// rebuilds.
|
||||
//
|
||||
// It is not free for the catalog. explore_index is a *downloaded
|
||||
// artifact*, not something derived from the user's files, and it is the
|
||||
// largest thing this app stores. So it went stale instead: plan 014
|
||||
// added `total_tracks` to the schema and to `indexRowFields` -- the one
|
||||
// projection every explore read uses -- and no database that already
|
||||
// existed ever grew the column. Every Explore search, browse, artist
|
||||
// page and album page on such an install fails with
|
||||
// "no such column: total_tracks", while a fresh install is perfectly
|
||||
// healthy, which is why the tests did not see it. The same databases
|
||||
// are stale a second way, from the same plan: their `mbid` columns are
|
||||
// still TEXT where the schema now declares BLOB, and SQLite does not
|
||||
// coerce between the two -- a comparison against 16 raw bytes simply
|
||||
// returns no rows.
|
||||
//
|
||||
// The repair is to notice and drop, not to migrate. A dropped catalog
|
||||
// costs one artifact download (about a minute); the alternative --
|
||||
// ALTER TABLE ADD COLUMN, which would handle `total_tracks` alone
|
||||
// cheaply -- cannot express the TEXT-to-BLOB half at all, and would
|
||||
// leave those installs quietly broken while reporting success.
|
||||
//
|
||||
// Everything except `Authored` is eligible. `Cache` is rebuildable by
|
||||
// definition; `Owned` is a projection of the user's files and a rescan
|
||||
// rebuilds it, which is plan 013's stated answer to exactly this
|
||||
// situation ("delete and rescan", open question 1); `Derived` is
|
||||
// computed from Owned. No `Authored` table is ever dropped here --
|
||||
// that is the whole point of the datamap, and it is asserted by
|
||||
// TestAuthoredTablesAreNeverRetired rather than only stated.
|
||||
//
|
||||
// What that does *not* buy is immunity for authored rows that reference
|
||||
// a retired table. `audio_files` is MIXED KIND: `play_count`,
|
||||
// `last_played` and `tag_status` are authored columns on an Owned
|
||||
// table, and they go with it. Playlists survive as playlists, and
|
||||
// their entries survive pointing at nothing. That cost was weighed and
|
||||
// accepted rather than overlooked -- the alternative is to carry the
|
||||
// authored columns across the rebuild keyed on file_path, which stays a
|
||||
// real option if this ever bites harder than it is worth.
|
||||
//
|
||||
// **This relies on foreign_keys being ON**, which applyPRAGMAs has
|
||||
// already done by the time NewDB calls it, and the dependency is not
|
||||
// cosmetic. SQLite performs an implicit DELETE before dropping a table
|
||||
// when foreign keys are enabled, so `playlist_tracks.audio_file_id` --
|
||||
// declared ON DELETE SET NULL -- is nulled. With foreign keys off, no
|
||||
// action fires and those rows keep the ids they had, which a rescan
|
||||
// then reissues starting from 1: every playlist would silently fill
|
||||
// with *different songs*. Nulled entries are merely empty; stale ones
|
||||
// are wrong, and wrong quietly. TestRetiringOwnedTablesDoesNotDangle
|
||||
// is what stops a future reordering turning one into the other.
|
||||
|
||||
// retireGroups are tables that must be retired together. A catalog
|
||||
// whose rows are gone must not keep the full-text index built over
|
||||
// them, nor the metadata claiming the import that produced them
|
||||
// finished -- that marker is exactly what stops the artifact being
|
||||
// fetched again. applySchema recreates all three empty immediately
|
||||
// afterwards, and the ordinary "no index yet" path takes over.
|
||||
var retireGroups = [][]string{
|
||||
{
|
||||
"explore_index",
|
||||
"explore_index_fts",
|
||||
"explore_index_meta",
|
||||
"explore_champion_fts",
|
||||
},
|
||||
}
|
||||
|
||||
// schemaColumn is one column as the schema file declares it.
|
||||
type schemaColumn struct {
|
||||
name string
|
||||
typ string
|
||||
}
|
||||
|
||||
// retireStaleTables drops every non-authored table whose live shape no
|
||||
// longer matches what sql/schemas/ declares, plus any table the schema
|
||||
// no longer describes at all, so applySchema can create the current
|
||||
// shape afresh. It runs before applySchema and is a no-op on a new
|
||||
// database, where the tables do not exist yet.
|
||||
func retireStaleTables(
|
||||
ctx context.Context, db *sql.DB, logger *slog.Logger,
|
||||
) error {
|
||||
declared, err := declaredTables()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
stale := make(map[string]string)
|
||||
|
||||
for table, columns := range declared {
|
||||
entry, ok := datamap.Lookup(table)
|
||||
if !ok || entry.Kind == datamap.Authored || entry.FTS {
|
||||
continue
|
||||
}
|
||||
|
||||
// Whether a stale Cache table may be rebuilt is decided per
|
||||
// binary, at compile time: the app re-downloads its catalog in
|
||||
// about a minute, cmd/indexbuild would re-derive it from ~205 GB
|
||||
// of dumps. See staleshape_policy.go.
|
||||
if entry.Kind == datamap.Cache && !retireStaleCache {
|
||||
continue
|
||||
}
|
||||
|
||||
reason, err := staleReason(ctx, db, table, columns)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if reason != "" {
|
||||
stale[table] = reason
|
||||
}
|
||||
}
|
||||
|
||||
obsolete, err := obsoleteTables(ctx, db)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
maps.Copy(stale, obsolete)
|
||||
|
||||
if len(stale) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
return retireGroupsFor(ctx, db, logger, stale)
|
||||
}
|
||||
|
||||
// obsoleteTables are live tables the schema no longer describes at all.
|
||||
// TestCatalogCoversSchema makes the datamap a complete description of
|
||||
// the current schema, so a table it does not know is one a past version
|
||||
// created and this one does not -- plan 013 alone left seven behind
|
||||
// (recordings, release_groups, artist_credit, artist_credit_artist,
|
||||
// release_group_recordings, recording_genres) plus the
|
||||
// schema_migrations table that squashing the chain retired. They are
|
||||
// dead weight, and one of them holding a foreign key into a table being
|
||||
// rebuilt is worse than dead weight.
|
||||
//
|
||||
// SQLite's own bookkeeping and FTS shadow tables are not obsolete:
|
||||
// datamap.Lookup resolves a shadow table to its parent, and IsInternal
|
||||
// covers the rest.
|
||||
func obsoleteTables(ctx context.Context, db *sql.DB) (map[string]string, error) {
|
||||
rows, err := db.QueryContext(
|
||||
ctx, "SELECT name FROM sqlite_master WHERE type = 'table'",
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("could not list tables: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
out := make(map[string]string)
|
||||
|
||||
for rows.Next() {
|
||||
var name string
|
||||
|
||||
if err := rows.Scan(&name); err != nil {
|
||||
return nil, fmt.Errorf("could not scan table name: %w", err)
|
||||
}
|
||||
|
||||
if datamap.IsInternal(name) {
|
||||
continue
|
||||
}
|
||||
|
||||
if _, known := datamap.Lookup(name); !known {
|
||||
out[name] = "the schema no longer describes this table"
|
||||
}
|
||||
}
|
||||
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, fmt.Errorf("could not read table list: %w", err)
|
||||
}
|
||||
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// retireGroupsFor drops each stale table along with everything its
|
||||
// retire group says must go with it.
|
||||
func retireGroupsFor(
|
||||
ctx context.Context, db *sql.DB, logger *slog.Logger,
|
||||
stale map[string]string,
|
||||
) error {
|
||||
drop := make(map[string]string)
|
||||
|
||||
for table, reason := range stale {
|
||||
drop[table] = reason
|
||||
|
||||
for _, group := range retireGroups {
|
||||
if !slices.Contains(group, table) {
|
||||
continue
|
||||
}
|
||||
|
||||
for _, member := range group {
|
||||
if _, already := drop[member]; !already {
|
||||
drop[member] = "retired with " + table
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return dropDeferred(ctx, db, logger, drop)
|
||||
}
|
||||
|
||||
// dropDeferred drops every named table in one transaction with foreign
|
||||
// key enforcement deferred to the commit.
|
||||
//
|
||||
// The deferral is required and the two obvious alternatives are both
|
||||
// wrong. These tables reference each other -- pre-013 `audio_files`
|
||||
// has a foreign key into `recordings`, which is itself being retired --
|
||||
// so dropping them one at a time in an arbitrary order fails with
|
||||
// "FOREIGN KEY constraint failed" on whichever is unlucky enough to go
|
||||
// first, and there is no order that is safe in general. Turning
|
||||
// foreign keys *off* for the duration would fix that and silently take
|
||||
// the ON DELETE SET NULL on `playlist_tracks.audio_file_id` with it,
|
||||
// leaving playlist entries pointing at ids a rescan reissues to
|
||||
// different songs -- the exact failure
|
||||
// TestRetiringOwnedTablesDoesNotDangle exists to prevent.
|
||||
//
|
||||
// Deferring keeps the actions firing while tolerating the inconsistency
|
||||
// in the middle, and the commit then checks that the end state is
|
||||
// sound. It is set inside the transaction because SQLite resets it at
|
||||
// every commit.
|
||||
func dropDeferred(
|
||||
ctx context.Context, db *sql.DB, logger *slog.Logger,
|
||||
drop map[string]string,
|
||||
) error {
|
||||
tx, err := db.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("could not begin the retire transaction: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = tx.Rollback() }()
|
||||
|
||||
if _, err := tx.ExecContext(ctx, "PRAGMA defer_foreign_keys = ON"); err != nil {
|
||||
return fmt.Errorf("could not defer foreign keys: %w", err)
|
||||
}
|
||||
|
||||
// Sorted, so a failure is reproducible. Map order is random, and a
|
||||
// bug that depends on which table happens to go first reproduces on
|
||||
// one run in three and passes review on the other two -- which is
|
||||
// exactly how the foreign-key ordering above reached a real
|
||||
// database. Sorting does not make any order *safe*; the deferral
|
||||
// does that.
|
||||
for _, table := range slices.Sorted(maps.Keys(drop)) {
|
||||
logger.Warn(
|
||||
"retiring a table the schema no longer describes",
|
||||
"table", table,
|
||||
"reason", drop[table],
|
||||
)
|
||||
|
||||
if _, err := tx.ExecContext(
|
||||
ctx, "DROP TABLE IF EXISTS "+quoteIdent(table),
|
||||
); err != nil {
|
||||
return fmt.Errorf("could not retire stale table %s: %w", table, err)
|
||||
}
|
||||
}
|
||||
|
||||
if err := tx.Commit(); err != nil {
|
||||
return fmt.Errorf("could not commit the retire: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// staleReason reports why a live table disagrees with its declaration,
|
||||
// or "" when it agrees. A column the live table does not have is the
|
||||
// additive case; a column whose declared type changed is the one an
|
||||
// ALTER could not fix anyway. Columns the live table has and the
|
||||
// schema no longer declares are ignored: they cost nothing and dropping
|
||||
// the table over one would retire a healthy catalog.
|
||||
func staleReason(
|
||||
ctx context.Context, db *sql.DB, table string, columns []schemaColumn,
|
||||
) (string, error) {
|
||||
live, err := liveColumns(ctx, db, table)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
if len(live) == 0 {
|
||||
// Not present at all: applySchema is about to create it.
|
||||
return "", nil
|
||||
}
|
||||
|
||||
for _, col := range columns {
|
||||
liveType, present := live[col.name]
|
||||
if !present {
|
||||
return "missing column " + col.name, nil
|
||||
}
|
||||
|
||||
if !sameDeclaredType(col.typ, liveType) {
|
||||
return fmt.Sprintf(
|
||||
"column %s is %s, schema declares %s",
|
||||
col.name, liveType, col.typ,
|
||||
), nil
|
||||
}
|
||||
}
|
||||
|
||||
return "", nil
|
||||
}
|
||||
|
||||
// liveColumns returns the live table's columns and their declared types,
|
||||
// empty when the table does not exist.
|
||||
func liveColumns(
|
||||
ctx context.Context, db *sql.DB, table string,
|
||||
) (map[string]string, error) {
|
||||
rows, err := db.QueryContext(
|
||||
ctx, "SELECT name, type FROM pragma_table_info(?)", table,
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("could not inspect table %s: %w", table, err)
|
||||
}
|
||||
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
out := make(map[string]string)
|
||||
|
||||
for rows.Next() {
|
||||
var name, typ string
|
||||
|
||||
if err := rows.Scan(&name, &typ); err != nil {
|
||||
return nil, fmt.Errorf("could not scan column of %s: %w", table, err)
|
||||
}
|
||||
|
||||
out[name] = typ
|
||||
}
|
||||
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, fmt.Errorf("could not read columns of %s: %w", table, err)
|
||||
}
|
||||
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// sameDeclaredType compares two SQLite type names. They are compared
|
||||
// case-insensitively and only on the leading word, so INTEGER matches
|
||||
// INTEGER and VARCHAR(20) matches VARCHAR -- SQLite's affinity rules
|
||||
// make finer distinctions meaningless, and a difference that fine is
|
||||
// not worth retiring a catalog over. An empty declared type matches
|
||||
// anything, which is what a column declared with only constraints has.
|
||||
func sameDeclaredType(declared, live string) bool {
|
||||
d := strings.ToUpper(strings.Fields(declared + " ")[0])
|
||||
l := strings.ToUpper(strings.Fields(live + " ")[0])
|
||||
|
||||
if d == "" || l == "" {
|
||||
return true
|
||||
}
|
||||
|
||||
if i := strings.IndexByte(d, '('); i >= 0 {
|
||||
d = d[:i]
|
||||
}
|
||||
|
||||
if i := strings.IndexByte(l, '('); i >= 0 {
|
||||
l = l[:i]
|
||||
}
|
||||
|
||||
return d == l
|
||||
}
|
||||
|
||||
// declaredTables parses every CREATE TABLE in sql/schemas/ into its
|
||||
// column list. Parsing the schema rather than writing the expectation
|
||||
// down a second time is the point: a second list is a second thing to
|
||||
// forget, which is the fault this whole file exists to repair.
|
||||
func declaredTables() (map[string][]schemaColumn, error) {
|
||||
dirEntries, err := schemas.ReadDir("sql/schemas")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("could not read schemas directory: %w", err)
|
||||
}
|
||||
|
||||
out := make(map[string][]schemaColumn)
|
||||
|
||||
for _, dirEntry := range dirEntries {
|
||||
if dirEntry.IsDir() {
|
||||
continue
|
||||
}
|
||||
|
||||
content, err := fs.ReadFile(schemas, path.Join("sql/schemas", dirEntry.Name()))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("could not read %s: %w", dirEntry.Name(), err)
|
||||
}
|
||||
|
||||
maps.Copy(out, parseCreateTables(string(content)))
|
||||
}
|
||||
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// constraintKeywords begin a table constraint rather than a column.
|
||||
var constraintKeywords = map[string]bool{
|
||||
"PRIMARY": true, "FOREIGN": true, "UNIQUE": true,
|
||||
"CHECK": true, "CONSTRAINT": true,
|
||||
}
|
||||
|
||||
// parseCreateTables extracts the column names and declared types of
|
||||
// every non-virtual CREATE TABLE in one schema file.
|
||||
func parseCreateTables(content string) map[string][]schemaColumn {
|
||||
out := make(map[string][]schemaColumn)
|
||||
rest := stripLineComments(content)
|
||||
|
||||
for {
|
||||
idx := indexFold(rest, "CREATE TABLE ")
|
||||
if idx < 0 {
|
||||
return out
|
||||
}
|
||||
|
||||
rest = rest[idx+len("CREATE TABLE "):]
|
||||
|
||||
head, body, ok := splitTableBody(rest)
|
||||
if !ok {
|
||||
return out
|
||||
}
|
||||
|
||||
if name := tableName(head); name != "" {
|
||||
out[name] = parseColumns(body)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// tableName pulls the table name out of the text between "CREATE TABLE"
|
||||
// and its opening parenthesis, dropping an IF NOT EXISTS and any
|
||||
// quoting.
|
||||
func tableName(head string) string {
|
||||
head = strings.TrimSpace(head)
|
||||
head = strings.TrimPrefix(head, "IF NOT EXISTS ")
|
||||
head = strings.TrimPrefix(head, "if not exists ")
|
||||
|
||||
fields := strings.Fields(head)
|
||||
if len(fields) == 0 {
|
||||
return ""
|
||||
}
|
||||
|
||||
return strings.Trim(fields[len(fields)-1], `"'`+"`")
|
||||
}
|
||||
|
||||
// splitTableBody returns the text before the table's opening paren and
|
||||
// the balanced text inside it.
|
||||
func splitTableBody(s string) (head, body string, ok bool) {
|
||||
open := strings.IndexByte(s, '(')
|
||||
if open < 0 {
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
depth := 0
|
||||
|
||||
for i := open; i < len(s); i++ {
|
||||
switch s[i] {
|
||||
case '(':
|
||||
depth++
|
||||
case ')':
|
||||
depth--
|
||||
|
||||
if depth == 0 {
|
||||
return s[:open], s[open+1 : i], true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
// parseColumns splits a table body on its top-level commas and keeps
|
||||
// the parts that are columns rather than table constraints.
|
||||
func parseColumns(body string) []schemaColumn {
|
||||
var (
|
||||
out []schemaColumn
|
||||
depth int
|
||||
start int
|
||||
)
|
||||
|
||||
parts := make([]string, 0, 8)
|
||||
|
||||
for i := range len(body) {
|
||||
switch body[i] {
|
||||
case '(':
|
||||
depth++
|
||||
case ')':
|
||||
depth--
|
||||
case ',':
|
||||
if depth == 0 {
|
||||
parts = append(parts, body[start:i])
|
||||
start = i + 1
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
parts = append(parts, body[start:])
|
||||
|
||||
for _, part := range parts {
|
||||
fields := strings.Fields(part)
|
||||
if len(fields) == 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
// A table constraint need not be followed by a space --
|
||||
// "UNIQUE(mbid)" is one field, and reading it as a column name
|
||||
// makes an entirely healthy table look stale, which retires a
|
||||
// catalog nobody asked to lose.
|
||||
head := fields[0]
|
||||
if i := strings.IndexByte(head, '('); i >= 0 {
|
||||
head = head[:i]
|
||||
}
|
||||
|
||||
if constraintKeywords[strings.ToUpper(head)] {
|
||||
continue
|
||||
}
|
||||
|
||||
col := schemaColumn{name: strings.Trim(head, `"'`+"`")}
|
||||
if len(fields) > 1 {
|
||||
col.typ = fields[1]
|
||||
}
|
||||
|
||||
out = append(out, col)
|
||||
}
|
||||
|
||||
return out
|
||||
}
|
||||
|
||||
// stripLineComments removes -- comments, which otherwise contribute
|
||||
// stray parentheses and commas to the parse.
|
||||
func stripLineComments(s string) string {
|
||||
lines := strings.Split(s, "\n")
|
||||
for i, line := range lines {
|
||||
if idx := strings.Index(line, "--"); idx >= 0 {
|
||||
lines[i] = line[:idx]
|
||||
}
|
||||
}
|
||||
|
||||
return strings.Join(lines, "\n")
|
||||
}
|
||||
|
||||
// indexFold is a case-insensitive strings.Index.
|
||||
func indexFold(s, substr string) int {
|
||||
return strings.Index(strings.ToUpper(s), strings.ToUpper(substr))
|
||||
}
|
||||
|
||||
// quoteIdent quotes a table name for interpolation into DDL, which
|
||||
// cannot take a bound parameter.
|
||||
func quoteIdent(name string) string {
|
||||
return `"` + strings.ReplaceAll(name, `"`, `""`) + `"`
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
//go:build !indexbuild
|
||||
|
||||
package database
|
||||
|
||||
// retireStaleCache reports whether a Cache table whose shape no longer
|
||||
// matches the schema may be dropped and rebuilt.
|
||||
//
|
||||
// In the app: yes. The only Cache table large enough to care about is
|
||||
// the catalog, and the app does not derive it — it downloads it. A
|
||||
// stale one costs about a minute of re-fetching the artifact, and
|
||||
// keeping it costs every Explore read on the install, because a
|
||||
// projection naming a column the table does not have fails outright.
|
||||
//
|
||||
// In cmd/indexbuild: no, and the file next to this one says why.
|
||||
const retireStaleCache = true
|
||||
@@ -0,0 +1,37 @@
|
||||
//go:build indexbuild
|
||||
|
||||
package database
|
||||
|
||||
// retireStaleCache is false here, and this is the whole reason the
|
||||
// policy is a build tag rather than a rule inside retireStaleTables.
|
||||
//
|
||||
// The index database is the one place in this project where the catalog
|
||||
// is *derived* rather than downloaded. Rebuilding it is a ~205 GB dump
|
||||
// stream over hours, resumed across runs from a checkpoint on a
|
||||
// persistent volume; that volume exists for no other purpose. The app's
|
||||
// answer to a stale catalog — drop it, fetch the artifact again — is
|
||||
// not available here, because this database *is* what the artifact is
|
||||
// cut from.
|
||||
//
|
||||
// This was not hypothetical. The repair shipped without it and dropped
|
||||
// the CI catalog on its first run:
|
||||
//
|
||||
// retiring a table ... table=explore_index
|
||||
// reason="column entity_type is TEXT, schema declares INTEGER"
|
||||
// index maintenance mode=build reason="no completed import yet"
|
||||
//
|
||||
// The shape mismatch was real and the drop was correct by the app's
|
||||
// rule. It was still wrong here: that database is deliberately kept in
|
||||
// the older encoding, which is what `fix(indexexport): read an index
|
||||
// older than the binary` exists to tolerate. A rule that is right for
|
||||
// every install and catastrophic for one database has to be told which
|
||||
// one it is in, and a build tag is how this project already tells the
|
||||
// index tools apart (backend/events/runtime_indexbuild.go,
|
||||
// backend/explore/servicestartup.go, dumpbuild_stub.go).
|
||||
//
|
||||
// cmd/indexbuild has its own repair for the half it *can* safely
|
||||
// discard: retireLibraryTables drops every table the datamap does not
|
||||
// classify as Cache, which is empty by construction in that database.
|
||||
// Between the two, the library half is repaired and the catalog is
|
||||
// never touched.
|
||||
const retireStaleCache = false
|
||||
@@ -0,0 +1,499 @@
|
||||
package database
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"log/slog"
|
||||
"path"
|
||||
"testing"
|
||||
|
||||
_ "modernc.org/sqlite"
|
||||
)
|
||||
|
||||
// testLogger discards the repair's warnings; the tests assert on the
|
||||
// database, not on the log.
|
||||
func testLogger() *slog.Logger {
|
||||
return slog.New(slog.DiscardHandler)
|
||||
}
|
||||
|
||||
// openRaw opens a scratch database file with no schema applied, so a
|
||||
// test can build an *old* shape and then let NewDB's repair meet it.
|
||||
func openRaw(t *testing.T, dir string) *sql.DB {
|
||||
t.Helper()
|
||||
|
||||
db, err := sql.Open("sqlite", path.Join(dir, "yj.db"))
|
||||
if err != nil {
|
||||
t.Fatalf("open: %v", err)
|
||||
}
|
||||
|
||||
t.Cleanup(func() { _ = db.Close() })
|
||||
|
||||
return db
|
||||
}
|
||||
|
||||
// TestRetiresIndexMissingAColumn is plan 014's bug, symptom first: an
|
||||
// explore_index created before `total_tracks` existed, met by the
|
||||
// projection every explore read uses. Before the repair this failed
|
||||
// with "no such column: total_tracks" on every install that already had
|
||||
// a catalog, while a fresh one was perfectly healthy.
|
||||
func TestRetiresIndexMissingAColumn(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
dir := t.TempDir()
|
||||
db := openRaw(t, dir)
|
||||
|
||||
// The pre-014 shape: the columns the projection needs, minus the
|
||||
// one the plan added.
|
||||
if _, err := db.ExecContext(ctx, `
|
||||
CREATE TABLE explore_index (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
entity_type INTEGER NOT NULL,
|
||||
mbid BLOB NOT NULL,
|
||||
title TEXT NOT NULL,
|
||||
artist_name TEXT NOT NULL,
|
||||
artist_mbid BLOB NOT NULL
|
||||
);
|
||||
INSERT INTO explore_index (entity_type, mbid, title, artist_name, artist_mbid)
|
||||
VALUES (1, x'00112233445566778899aabbccddeeff', 'x', 'y', x'');
|
||||
`); err != nil {
|
||||
t.Fatalf("seed: %v", err)
|
||||
}
|
||||
|
||||
if err := retireStaleTables(ctx, db, testLogger()); err != nil {
|
||||
t.Fatalf("retire: %v", err)
|
||||
}
|
||||
|
||||
if err := applySchema(ctx, db); err != nil {
|
||||
t.Fatalf("applySchema: %v", err)
|
||||
}
|
||||
|
||||
// The column the projection needs is there now.
|
||||
var n int
|
||||
if err := db.QueryRowContext(ctx,
|
||||
`SELECT COUNT(*) FROM pragma_table_info('explore_index')
|
||||
WHERE name = 'total_tracks'`,
|
||||
).Scan(&n); err != nil {
|
||||
t.Fatalf("inspect: %v", err)
|
||||
}
|
||||
|
||||
if n != 1 {
|
||||
t.Fatalf("explore_index still has no total_tracks column")
|
||||
}
|
||||
|
||||
// And the catalog really was retired rather than patched, so the
|
||||
// artifact is fetched again instead of half a catalog being served.
|
||||
if err := db.QueryRowContext(ctx,
|
||||
"SELECT COUNT(*) FROM explore_index",
|
||||
).Scan(&n); err != nil {
|
||||
t.Fatalf("count: %v", err)
|
||||
}
|
||||
|
||||
if n != 0 {
|
||||
t.Fatalf("stale rows survived the retire: %d", n)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRetiresIndexWithTextMBIDs is the half an ALTER could not have
|
||||
// repaired: plan 013 changed mbid from TEXT to BLOB, and SQLite does not
|
||||
// coerce between them, so a query against 16 raw bytes returns no rows
|
||||
// rather than an error.
|
||||
func TestRetiresIndexWithTextMBIDs(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
dir := t.TempDir()
|
||||
db := openRaw(t, dir)
|
||||
|
||||
if _, err := db.ExecContext(ctx, `
|
||||
CREATE TABLE explore_index (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
entity_type TEXT NOT NULL,
|
||||
mbid TEXT NOT NULL,
|
||||
title TEXT NOT NULL,
|
||||
artist_name TEXT NOT NULL,
|
||||
artist_mbid TEXT NOT NULL,
|
||||
total_tracks INTEGER NOT NULL DEFAULT 0
|
||||
);
|
||||
`); err != nil {
|
||||
t.Fatalf("seed: %v", err)
|
||||
}
|
||||
|
||||
if err := retireStaleTables(ctx, db, testLogger()); err != nil {
|
||||
t.Fatalf("retire: %v", err)
|
||||
}
|
||||
|
||||
if err := applySchema(ctx, db); err != nil {
|
||||
t.Fatalf("applySchema: %v", err)
|
||||
}
|
||||
|
||||
var typ string
|
||||
if err := db.QueryRowContext(ctx,
|
||||
`SELECT type FROM pragma_table_info('explore_index') WHERE name = 'mbid'`,
|
||||
).Scan(&typ); err != nil {
|
||||
t.Fatalf("inspect: %v", err)
|
||||
}
|
||||
|
||||
if typ != "BLOB" {
|
||||
t.Fatalf("mbid is still %s, want BLOB", typ)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRetiringTheIndexTakesItsMetaWithIt guards the thing that makes the
|
||||
// repair actually repair: the marker saying the import finished is what
|
||||
// stops the artifact being fetched again, so a catalog dropped without
|
||||
// it would stay empty forever.
|
||||
func TestRetiringTheIndexTakesItsMetaWithIt(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
dir := t.TempDir()
|
||||
db := openRaw(t, dir)
|
||||
|
||||
if _, err := db.ExecContext(ctx, `
|
||||
CREATE TABLE explore_index (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
entity_type INTEGER NOT NULL,
|
||||
mbid BLOB NOT NULL
|
||||
);
|
||||
CREATE TABLE explore_index_meta (key TEXT PRIMARY KEY, value TEXT NOT NULL);
|
||||
INSERT INTO explore_index_meta VALUES ('dump_import_done', '1');
|
||||
`); err != nil {
|
||||
t.Fatalf("seed: %v", err)
|
||||
}
|
||||
|
||||
if err := retireStaleTables(ctx, db, testLogger()); err != nil {
|
||||
t.Fatalf("retire: %v", err)
|
||||
}
|
||||
|
||||
if err := applySchema(ctx, db); err != nil {
|
||||
t.Fatalf("applySchema: %v", err)
|
||||
}
|
||||
|
||||
var n int
|
||||
if err := db.QueryRowContext(ctx,
|
||||
"SELECT COUNT(*) FROM explore_index_meta WHERE key = 'dump_import_done'",
|
||||
).Scan(&n); err != nil {
|
||||
t.Fatalf("meta: %v", err)
|
||||
}
|
||||
|
||||
if n != 0 {
|
||||
t.Fatalf("the import-done marker survived a retired catalog")
|
||||
}
|
||||
}
|
||||
|
||||
// TestHealthyDatabaseIsUntouched is the other half, and the one that
|
||||
// would make this dangerous if it failed: a current schema must survive
|
||||
// a launch with its catalog intact. A repair that retires a healthy
|
||||
// catalog costs every user an artifact download on every start.
|
||||
func TestHealthyDatabaseIsUntouched(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
dir := t.TempDir()
|
||||
db := openRaw(t, dir)
|
||||
|
||||
if err := applySchema(ctx, db); err != nil {
|
||||
t.Fatalf("applySchema: %v", err)
|
||||
}
|
||||
|
||||
if _, err := db.ExecContext(ctx, `
|
||||
INSERT INTO explore_index (entity_type, mbid, title, artist_name, artist_mbid)
|
||||
VALUES (1, x'00112233445566778899aabbccddeeff', 'x', 'y', x'')
|
||||
`); err != nil {
|
||||
t.Fatalf("seed: %v", err)
|
||||
}
|
||||
|
||||
if err := retireStaleTables(ctx, db, testLogger()); err != nil {
|
||||
t.Fatalf("retire: %v", err)
|
||||
}
|
||||
|
||||
var n int
|
||||
if err := db.QueryRowContext(ctx,
|
||||
"SELECT COUNT(*) FROM explore_index",
|
||||
).Scan(&n); err != nil {
|
||||
t.Fatalf("count: %v", err)
|
||||
}
|
||||
|
||||
if n != 1 {
|
||||
t.Fatalf("a healthy catalog was retired: %d rows left", n)
|
||||
}
|
||||
}
|
||||
|
||||
// TestAuthoredTablesAreNeverRetired states the boundary in a test rather
|
||||
// than only in a comment: this mechanism deletes data, and the only
|
||||
// thing standing between it and a user's playlists is the Kind filter.
|
||||
func TestAuthoredTablesAreNeverRetired(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
dir := t.TempDir()
|
||||
db := openRaw(t, dir)
|
||||
|
||||
// A playlists table missing most of its current columns.
|
||||
if _, err := db.ExecContext(ctx, `
|
||||
CREATE TABLE playlists (id INTEGER PRIMARY KEY, name TEXT NOT NULL);
|
||||
INSERT INTO playlists (name) VALUES ('irreplaceable');
|
||||
`); err != nil {
|
||||
t.Fatalf("seed: %v", err)
|
||||
}
|
||||
|
||||
if err := retireStaleTables(ctx, db, testLogger()); err != nil {
|
||||
t.Fatalf("retire: %v", err)
|
||||
}
|
||||
|
||||
var n int
|
||||
if err := db.QueryRowContext(ctx,
|
||||
"SELECT COUNT(*) FROM playlists",
|
||||
).Scan(&n); err != nil {
|
||||
t.Fatalf("count: %v", err)
|
||||
}
|
||||
|
||||
if n != 1 {
|
||||
t.Fatalf("an authored table was retired; rows left: %d", n)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRetiresTablesTheSchemaNoLongerDescribes covers what plan 013 left
|
||||
// behind on every database that predates it: seven tables the schema
|
||||
// stopped describing, plus the schema_migrations table that squashing
|
||||
// the chain retired. They are not stale in shape — they are simply not
|
||||
// ours any more.
|
||||
func TestRetiresTablesTheSchemaNoLongerDescribes(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
db := openRaw(t, t.TempDir())
|
||||
|
||||
if _, err := db.ExecContext(ctx, `
|
||||
CREATE TABLE recordings (id INTEGER PRIMARY KEY, name TEXT);
|
||||
CREATE TABLE artist_credit (id INTEGER PRIMARY KEY, text TEXT);
|
||||
CREATE TABLE schema_migrations (version INTEGER PRIMARY KEY);
|
||||
`); err != nil {
|
||||
t.Fatalf("seed: %v", err)
|
||||
}
|
||||
|
||||
if err := retireStaleTables(ctx, db, testLogger()); err != nil {
|
||||
t.Fatalf("retire: %v", err)
|
||||
}
|
||||
|
||||
for _, table := range []string{"recordings", "artist_credit", "schema_migrations"} {
|
||||
var n int
|
||||
|
||||
if err := db.QueryRowContext(ctx,
|
||||
"SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name = ?",
|
||||
table,
|
||||
).Scan(&n); err != nil {
|
||||
t.Fatalf("inspect %s: %v", table, err)
|
||||
}
|
||||
|
||||
if n != 0 {
|
||||
t.Errorf("%s survived; the schema no longer describes it", table)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestFTSShadowTablesAreNotObsolete is the sweep's sharp edge: an FTS5
|
||||
// virtual table is backed by four shadow tables that appear in
|
||||
// sqlite_master under their own names and are in no schema file.
|
||||
// Dropping one destroys the index it belongs to.
|
||||
func TestFTSShadowTablesAreNotObsolete(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
db := openRaw(t, t.TempDir())
|
||||
|
||||
if err := applySchema(ctx, db); err != nil {
|
||||
t.Fatalf("applySchema: %v", err)
|
||||
}
|
||||
|
||||
obsolete, err := obsoleteTables(ctx, db)
|
||||
if err != nil {
|
||||
t.Fatalf("obsoleteTables: %v", err)
|
||||
}
|
||||
|
||||
if len(obsolete) != 0 {
|
||||
t.Fatalf("a freshly created schema reported obsolete tables: %v", obsolete)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRetiringOwnedTablesDoesNotDangle pins the one behaviour that is
|
||||
// silently wrong rather than loudly broken.
|
||||
//
|
||||
// Retiring audio_files leaves playlist entries behind. With
|
||||
// foreign_keys ON — which applyPRAGMAs has done before NewDB gets here —
|
||||
// SET NULL fires and they point at nothing. With it OFF they keep ids
|
||||
// that the rescan reissues from 1, so every playlist quietly fills with
|
||||
// different songs. Nothing about the schema makes that ordering
|
||||
// obvious, so it is asserted rather than assumed.
|
||||
func TestRetiringOwnedTablesDoesNotDangle(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
db := openRaw(t, t.TempDir())
|
||||
|
||||
if _, err := db.ExecContext(ctx, "PRAGMA foreign_keys = ON"); err != nil {
|
||||
t.Fatalf("pragma: %v", err)
|
||||
}
|
||||
|
||||
if err := applySchema(ctx, db); err != nil {
|
||||
t.Fatalf("applySchema: %v", err)
|
||||
}
|
||||
|
||||
// Break audio_files' shape so it is retired, keeping a playlist
|
||||
// entry that references it.
|
||||
if _, err := db.ExecContext(ctx, `
|
||||
INSERT INTO playlists (id, name) VALUES (1, 'keepme');
|
||||
INSERT INTO libraries (id, name, path) VALUES (0, 'test', '/music');
|
||||
INSERT INTO audio_files (id, file_path, file_type_id, length_milliseconds)
|
||||
VALUES (7, '/music/a.flac', 1, 1000);
|
||||
INSERT INTO playlist_tracks (playlist_id, audio_file_id, position)
|
||||
VALUES (1, 7, 0);
|
||||
DROP VIEW IF EXISTS track_metadata;
|
||||
ALTER TABLE audio_files DROP COLUMN artist_credit;
|
||||
`); err != nil {
|
||||
t.Fatalf("seed: %v", err)
|
||||
}
|
||||
|
||||
if err := retireStaleTables(ctx, db, testLogger()); err != nil {
|
||||
t.Fatalf("retire: %v", err)
|
||||
}
|
||||
|
||||
if err := applySchema(ctx, db); err != nil {
|
||||
t.Fatalf("applySchema: %v", err)
|
||||
}
|
||||
|
||||
var dangling int
|
||||
if err := db.QueryRowContext(ctx,
|
||||
"SELECT COUNT(*) FROM playlist_tracks WHERE audio_file_id IS NOT NULL",
|
||||
).Scan(&dangling); err != nil {
|
||||
t.Fatalf("count: %v", err)
|
||||
}
|
||||
|
||||
if dangling != 0 {
|
||||
t.Fatalf(
|
||||
"%d playlist entries still point at retired audio_files ids; "+
|
||||
"a rescan will reissue those ids to different tracks",
|
||||
dangling,
|
||||
)
|
||||
}
|
||||
|
||||
// The playlist itself is authored and must be untouched.
|
||||
var playlists int
|
||||
if err := db.QueryRowContext(ctx,
|
||||
"SELECT COUNT(*) FROM playlists",
|
||||
).Scan(&playlists); err != nil {
|
||||
t.Fatalf("playlists: %v", err)
|
||||
}
|
||||
|
||||
if playlists != 1 {
|
||||
t.Fatalf("authored playlist lost: %d", playlists)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRetiringInterlinkedLegacyTables is the bug the unit tests missed
|
||||
// and a real database found.
|
||||
//
|
||||
// The tables plan 013 retired reference each other -- pre-013
|
||||
// audio_files has a foreign key into recordings -- so with foreign keys
|
||||
// ON, dropping them one at a time fails with "FOREIGN KEY constraint
|
||||
// failed" on whichever goes first, and map iteration order decides
|
||||
// which that is. Every other test in this file ran with foreign keys
|
||||
// off and passed happily; the app enables them in applyPRAGMAs before
|
||||
// the repair runs, so only the real launch path showed it.
|
||||
func TestRetiringInterlinkedLegacyTables(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
db := openRaw(t, t.TempDir())
|
||||
|
||||
if _, err := db.ExecContext(ctx, "PRAGMA foreign_keys = ON"); err != nil {
|
||||
t.Fatalf("pragma: %v", err)
|
||||
}
|
||||
|
||||
// The pre-013 shape, with the reference that makes ordering matter.
|
||||
// release_group_recordings sorts *after* recordings and references
|
||||
// it, so the deterministic order retires the parent while the child
|
||||
// still holds rows pointing at it -- which is the case that fails
|
||||
// without the deferral, rather than one that fails on some runs.
|
||||
if _, err := db.ExecContext(ctx, `
|
||||
CREATE TABLE recordings (id INTEGER PRIMARY KEY, name TEXT);
|
||||
CREATE TABLE artist_credit (id INTEGER PRIMARY KEY, text TEXT);
|
||||
CREATE TABLE release_group_recordings (
|
||||
id INTEGER PRIMARY KEY,
|
||||
recording_id INTEGER NOT NULL,
|
||||
FOREIGN KEY(recording_id) REFERENCES recordings(id)
|
||||
);
|
||||
CREATE TABLE audio_files (
|
||||
id INTEGER PRIMARY KEY,
|
||||
file_path TEXT NOT NULL UNIQUE,
|
||||
recording_id INTEGER,
|
||||
FOREIGN KEY(recording_id) REFERENCES recordings(id)
|
||||
);
|
||||
INSERT INTO recordings (id, name) VALUES (1, 'x');
|
||||
INSERT INTO release_group_recordings (id, recording_id) VALUES (1, 1);
|
||||
INSERT INTO audio_files (id, file_path, recording_id)
|
||||
VALUES (1, '/music/a.flac', 1);
|
||||
`); err != nil {
|
||||
t.Fatalf("seed: %v", err)
|
||||
}
|
||||
|
||||
if err := retireStaleTables(ctx, db, testLogger()); err != nil {
|
||||
t.Fatalf("retire: %v", err)
|
||||
}
|
||||
|
||||
if err := applySchema(ctx, db); err != nil {
|
||||
t.Fatalf("applySchema: %v", err)
|
||||
}
|
||||
|
||||
for _, table := range []string{"recordings", "artist_credit"} {
|
||||
var n int
|
||||
|
||||
if err := db.QueryRowContext(ctx,
|
||||
"SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name = ?",
|
||||
table,
|
||||
).Scan(&n); err != nil {
|
||||
t.Fatalf("inspect %s: %v", table, err)
|
||||
}
|
||||
|
||||
if n != 0 {
|
||||
t.Errorf("%s survived the retire", table)
|
||||
}
|
||||
}
|
||||
|
||||
// And the rebuilt audio_files is the current shape, which is the
|
||||
// whole reason the old one had to go.
|
||||
var n int
|
||||
if err := db.QueryRowContext(ctx,
|
||||
`SELECT COUNT(*) FROM pragma_table_info('audio_files')
|
||||
WHERE name = 'artist_credit'`,
|
||||
).Scan(&n); err != nil {
|
||||
t.Fatalf("inspect audio_files: %v", err)
|
||||
}
|
||||
|
||||
if n != 1 {
|
||||
t.Fatal("audio_files was not rebuilt in the current shape")
|
||||
}
|
||||
}
|
||||
|
||||
// TestParseCreateTablesReadsTheRealSchema keeps the parser honest
|
||||
// against the files it actually runs on: a parser that silently found
|
||||
// no columns would report every table healthy and repair nothing.
|
||||
func TestParseCreateTablesReadsTheRealSchema(t *testing.T) {
|
||||
declared, err := declaredTables()
|
||||
if err != nil {
|
||||
t.Fatalf("declaredTables: %v", err)
|
||||
}
|
||||
|
||||
cols, ok := declared["explore_index"]
|
||||
if !ok {
|
||||
t.Fatal("explore_index was not parsed out of the schema files")
|
||||
}
|
||||
|
||||
want := map[string]string{
|
||||
"mbid": "BLOB",
|
||||
"total_tracks": "INTEGER",
|
||||
"artist_name": "TEXT",
|
||||
}
|
||||
|
||||
got := make(map[string]string, len(cols))
|
||||
for _, c := range cols {
|
||||
got[c.name] = c.typ
|
||||
}
|
||||
|
||||
for name, typ := range want {
|
||||
if got[name] != typ {
|
||||
t.Errorf("explore_index.%s parsed as %q, want %q", name, got[name], typ)
|
||||
}
|
||||
}
|
||||
|
||||
// A table constraint must not be mistaken for a column.
|
||||
for _, c := range cols {
|
||||
switch c.name {
|
||||
case "PRIMARY", "FOREIGN", "UNIQUE", "CHECK", "CONSTRAINT":
|
||||
t.Errorf("parsed table constraint %q as a column", c.name)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -185,6 +185,21 @@ var tables = []Table{
|
||||
Note: "Full-text index over the champion entities of the " +
|
||||
"MusicBrainz dump. Rebuilt only by a full index build.",
|
||||
},
|
||||
{
|
||||
Name: "artist_credit_part", Kind: Cache, Lifetime: Retained,
|
||||
Note: "The decomposition of a multi-artist credit, from the " +
|
||||
"MusicBrainz dump: one row per credited artist, with the " +
|
||||
"name as credited and the join phrase that follows it. " +
|
||||
"Arrives with the downloaded artifact, so rebuilding it " +
|
||||
"costs a dump stream and it is never swept.",
|
||||
},
|
||||
{
|
||||
Name: "artist_credit_ref", Kind: Cache, Lifetime: Retained,
|
||||
Note: "Which credit a catalog recording or release group is " +
|
||||
"credited to. Present only for multi-artist credits; " +
|
||||
"absence means one artist, which explore_index already " +
|
||||
"describes. Ships and dies with artist_credit_part.",
|
||||
},
|
||||
{
|
||||
Name: "explore_index", Kind: Cache, Lifetime: Retained,
|
||||
Note: "The offline MusicBrainz search index. Rebuilding costs a " +
|
||||
|
||||
@@ -199,24 +199,23 @@ func TestManagerEndToEndAutoPick(t *testing.T) {
|
||||
t.Errorf("expected imported file at %s: %v", want, err)
|
||||
}
|
||||
|
||||
// Staging was released only after a successful import.
|
||||
entries, err := os.ReadDir(f.staging.Root())
|
||||
if err != nil {
|
||||
t.Fatalf("read staging root: %v", err)
|
||||
}
|
||||
// Staging release and the rescan happen *after* the state is
|
||||
// recorded (manager.go sets StateComplete, then releases, then
|
||||
// scans), so waiting on the state is not waiting on these. Under
|
||||
// load the worker is descheduled in between and asserting straight
|
||||
// away reads the world one step too early -- which is exactly how
|
||||
// this test failed on a busy machine while passing alone.
|
||||
waitFor(t, func() bool {
|
||||
entries, err := os.ReadDir(f.staging.Root())
|
||||
if err != nil || len(entries) != 0 {
|
||||
return false
|
||||
}
|
||||
|
||||
if len(entries) != 0 {
|
||||
t.Errorf("staging not released: %d dirs remain", len(entries))
|
||||
}
|
||||
f.lib.mu.Lock()
|
||||
defer f.lib.mu.Unlock()
|
||||
|
||||
// The library was told to rescan.
|
||||
f.lib.mu.Lock()
|
||||
scanned := len(f.lib.scanned)
|
||||
f.lib.mu.Unlock()
|
||||
|
||||
if scanned != 1 {
|
||||
t.Errorf("library scans = %d, want 1", scanned)
|
||||
}
|
||||
return len(f.lib.scanned) == 1
|
||||
}, "staging was never released, or the library was never rescanned")
|
||||
}
|
||||
|
||||
// An ambiguous result set must park for the user rather than guess.
|
||||
|
||||
@@ -20,6 +20,30 @@ func newServiceFixture(t *testing.T) serviceFixture {
|
||||
mf := newManagerFixture(t)
|
||||
svc := NewService(slogDiscard(), mf.manager, mf.store, NewMemSecretStore())
|
||||
|
||||
// Every test here is about the durable Request that `StartDownload`
|
||||
// leaves behind, and none of them is about the download itself -- but
|
||||
// their fixture is an anchored four-track request with a healthy
|
||||
// provider, which is exactly what `AutoPickable` says yes to. So
|
||||
// `Manager.Start` was firing `go m.grab(...)`, detached and with
|
||||
// `context.WithoutCancel`, and the test then raced it.
|
||||
//
|
||||
// It lost, twice, in CI (`check` on c03c0b8, and nowhere locally):
|
||||
//
|
||||
// service_test.go:66: state = "satisfied", want wanted
|
||||
// testing.go:1369: TempDir RemoveAll cleanup: ... directory not empty
|
||||
//
|
||||
// The first is the request reaching its *next* state before the
|
||||
// assertion read it; the second is that same goroutine still writing
|
||||
// into `t.TempDir()` after the test returned. One cause, two shapes.
|
||||
//
|
||||
// Putting the candidate outside the auto-pick size window stops the
|
||||
// grab from ever starting, which is better than waiting for it: there
|
||||
// is no goroutine to be slow, so the tests state what they mean
|
||||
// ("the request exists, in this state") without a timing assumption
|
||||
// underneath. A test that does want the download has `managerFixture`
|
||||
// and sets its own preferences.
|
||||
mf.manager.SetPreferences(AutoDownloadPrefs{MaxSizeMB: 1})
|
||||
|
||||
return serviceFixture{managerFixture: mf, svc: svc}
|
||||
}
|
||||
|
||||
|
||||
@@ -283,6 +283,9 @@ func (si *SearchIndex) importCoreArtifact(ctx context.Context, path string) erro
|
||||
}
|
||||
|
||||
merged, mergeErr := si.mergeArtifactRows(ctx, info.rows)
|
||||
if mergeErr == nil {
|
||||
si.mergeArtifactCredits(ctx)
|
||||
}
|
||||
|
||||
if ftsSuspended {
|
||||
start := time.Now()
|
||||
@@ -473,3 +476,75 @@ func (si *SearchIndex) removeArtifactFile(path string) {
|
||||
si.logger.Warn("core artifact: cleanup failed", "path", path, "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
// artifactHasCredits reports whether the attached artifact carries the
|
||||
// multi-artist credit tables.
|
||||
//
|
||||
// The same shape, and the same handle, as artifactHasTotals above: an
|
||||
// artifact published before credits existed is still a perfectly good
|
||||
// catalog, and there is one already out there. Selecting from a table
|
||||
// that is not in it would fail an import that should have succeeded, so
|
||||
// it is asked rather than assumed -- on the *writer*, because `core` is
|
||||
// attached to that one connection and the read pool cannot see it.
|
||||
func (si *SearchIndex) artifactHasCredits() bool {
|
||||
var n int
|
||||
|
||||
err := si.db.QueryRowWriter(
|
||||
`SELECT COUNT(*) FROM core.sqlite_master
|
||||
WHERE type = 'table' AND name IN ('artist_credit_part', 'artist_credit_ref')`,
|
||||
).Scan(&n)
|
||||
|
||||
return err == nil && n == 2
|
||||
}
|
||||
|
||||
// mergeArtifactCredits copies the credit decomposition out of the
|
||||
// attached artifact.
|
||||
//
|
||||
// Credits are replaced wholesale rather than merged: they are derived
|
||||
// entirely from one dump build, they are keyed by ids that are only
|
||||
// meaningful within the artifact that carried them, and a half-updated
|
||||
// credit renders as the wrong artists rather than as missing ones.
|
||||
//
|
||||
// A failure here is logged and not returned. The catalog has already
|
||||
// merged at this point, and a catalog without credits is the catalog
|
||||
// this app had before them -- every credit falls back to its single
|
||||
// artist, which is the same fallback an untagged file already gets.
|
||||
func (si *SearchIndex) mergeArtifactCredits(ctx context.Context) {
|
||||
if !si.artifactHasCredits() {
|
||||
si.logger.Info("core artifact: no credit tables, keeping single-artist credits")
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
start := time.Now()
|
||||
|
||||
for _, stmt := range []string{
|
||||
"DELETE FROM artist_credit_part",
|
||||
"DELETE FROM artist_credit_ref",
|
||||
`INSERT OR REPLACE INTO artist_credit_part
|
||||
(credit_id, position, artist_mbid, credited_name, join_phrase)
|
||||
SELECT credit_id, position, artist_mbid, credited_name, join_phrase
|
||||
FROM core.artist_credit_part`,
|
||||
`INSERT OR REPLACE INTO artist_credit_ref (mbid, credit_id)
|
||||
SELECT mbid, credit_id FROM core.artist_credit_ref`,
|
||||
} {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return
|
||||
}
|
||||
|
||||
if _, err := si.db.ExecContext(stmt); err != nil {
|
||||
si.logger.Warn("core artifact: credit merge failed", "error", err)
|
||||
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
var refs int
|
||||
|
||||
_ = si.db.QueryRowWriter("SELECT COUNT(*) FROM artist_credit_ref").Scan(&refs)
|
||||
|
||||
si.logger.Info("core artifact: credits merged",
|
||||
"entities", refs,
|
||||
"elapsed", time.Since(start).Round(time.Millisecond),
|
||||
)
|
||||
}
|
||||
|
||||
@@ -3,6 +3,7 @@ package explore
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/hex"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
@@ -597,3 +598,153 @@ func TestImportCoreArtifactReadsTotalsWhenPresent(t *testing.T) {
|
||||
t.Errorf("TotalTracks = %d, want 0 (the catalog does not say)", old.TotalTracks)
|
||||
}
|
||||
}
|
||||
|
||||
// addArtifactCredits gives an artifact file the credit tables the
|
||||
// exporter now writes, so the import path can be exercised against one
|
||||
// that has them.
|
||||
func addArtifactCredits(t *testing.T, path string) {
|
||||
t.Helper()
|
||||
|
||||
db, err := sql.Open("sqlite", "file:"+path)
|
||||
if err != nil {
|
||||
t.Fatalf("open artifact: %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = db.Close() }()
|
||||
|
||||
for _, stmt := range []string{
|
||||
`CREATE TABLE artist_credit_part (
|
||||
credit_id INTEGER NOT NULL,
|
||||
position INTEGER NOT NULL,
|
||||
artist_mbid BLOB NOT NULL,
|
||||
credited_name TEXT NOT NULL,
|
||||
join_phrase TEXT NOT NULL DEFAULT '',
|
||||
PRIMARY KEY (credit_id, position)
|
||||
) WITHOUT ROWID`,
|
||||
`CREATE TABLE artist_credit_ref (
|
||||
mbid BLOB NOT NULL PRIMARY KEY,
|
||||
credit_id INTEGER NOT NULL
|
||||
) WITHOUT ROWID`,
|
||||
} {
|
||||
if _, err := db.Exec(stmt); err != nil {
|
||||
t.Fatalf("create credit tables: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// The packed form the catalog stores. uuid16/parseUUID live behind
|
||||
// the indexbuild tag, so this file decodes for itself.
|
||||
pack := func(mbid string) []byte {
|
||||
raw, err := hex.DecodeString(strings.ReplaceAll(mbid, "-", ""))
|
||||
if err != nil || len(raw) != 16 {
|
||||
t.Fatalf("fixture MBID %q is not a UUID: %v", mbid, err)
|
||||
}
|
||||
|
||||
return raw
|
||||
}
|
||||
|
||||
a, b, rec := pack(artA), pack(artB), pack(recA)
|
||||
|
||||
for _, part := range [][]any{
|
||||
{7, 0, a, "Artist A", " feat. "},
|
||||
{7, 1, b, "Artist B", ""},
|
||||
} {
|
||||
if _, err := db.Exec(`INSERT INTO artist_credit_part
|
||||
(credit_id, position, artist_mbid, credited_name, join_phrase)
|
||||
VALUES (?, ?, ?, ?, ?)`, part...); err != nil {
|
||||
t.Fatalf("insert part: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
if _, err := db.Exec(
|
||||
"INSERT INTO artist_credit_ref (mbid, credit_id) VALUES (?, ?)", rec, 7,
|
||||
); err != nil {
|
||||
t.Fatalf("insert ref: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestImportCoreArtifactMergesCredits is the positive half of the
|
||||
// compatibility pair: an artifact that carries credits delivers them,
|
||||
// rendering back to the credit string they decompose.
|
||||
func TestImportCoreArtifactMergesCredits(t *testing.T) {
|
||||
db := database.NewTestDB(t)
|
||||
si := NewSearchIndex(db, nil, nil, testLogger())
|
||||
|
||||
path := writeTestArtifact(t, validMeta(), []artifactRow{
|
||||
{"recording", recA, "Song A", "Artist A feat. Artist B", artA, 2000},
|
||||
})
|
||||
|
||||
addArtifactCredits(t, path)
|
||||
|
||||
if err := si.importCoreArtifact(context.Background(), path); err != nil {
|
||||
t.Fatalf("importCoreArtifact: %v", err)
|
||||
}
|
||||
|
||||
rows, err := db.QueryContext(
|
||||
`SELECT p.credited_name, p.join_phrase
|
||||
FROM artist_credit_ref r
|
||||
JOIN artist_credit_part p ON p.credit_id = r.credit_id
|
||||
ORDER BY p.position`,
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("query credits: %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
var rendered strings.Builder
|
||||
|
||||
for rows.Next() {
|
||||
var name, join string
|
||||
|
||||
if err := rows.Scan(&name, &join); err != nil {
|
||||
t.Fatalf("scan: %v", err)
|
||||
}
|
||||
|
||||
rendered.WriteString(name)
|
||||
rendered.WriteString(join)
|
||||
}
|
||||
|
||||
if got := rendered.String(); got != "Artist A feat. Artist B" {
|
||||
t.Errorf("rendered credit = %q, want %q", got, "Artist A feat. Artist B")
|
||||
}
|
||||
}
|
||||
|
||||
// TestImportCoreArtifactWithoutCredits is the regression that matters
|
||||
// most here: an artifact published before credits existed cannot be
|
||||
// re-cut retroactively, so it must import as a catalog that declines to
|
||||
// answer rather than failing outright. writeTestArtifact deliberately
|
||||
// builds one without the tables.
|
||||
func TestImportCoreArtifactWithoutCredits(t *testing.T) {
|
||||
db := database.NewTestDB(t)
|
||||
si := NewSearchIndex(db, nil, nil, testLogger())
|
||||
|
||||
path := writeTestArtifact(t, validMeta(), []artifactRow{
|
||||
{"recording", recA, "Song A", "Artist A", artA, 2000},
|
||||
})
|
||||
|
||||
if err := si.importCoreArtifact(context.Background(), path); err != nil {
|
||||
t.Fatalf("an artifact without credit tables must still import: %v", err)
|
||||
}
|
||||
|
||||
var rows int
|
||||
if err := db.QueryRowWriter(
|
||||
"SELECT COUNT(*) FROM explore_index",
|
||||
).Scan(&rows); err != nil {
|
||||
t.Fatalf("count: %v", err)
|
||||
}
|
||||
|
||||
if rows != 1 {
|
||||
t.Errorf("catalog rows = %d, want 1", rows)
|
||||
}
|
||||
|
||||
var refs int
|
||||
if err := db.QueryRowWriter(
|
||||
"SELECT COUNT(*) FROM artist_credit_ref",
|
||||
).Scan(&refs); err != nil {
|
||||
t.Fatalf("count refs: %v", err)
|
||||
}
|
||||
|
||||
if refs != 0 {
|
||||
t.Errorf("credit refs = %d, want 0", refs)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,132 @@
|
||||
package explore
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Reading multi-artist credits back out of the catalog.
|
||||
//
|
||||
// The tables are filled centrally (backend/explore/dumpcredits.go, and
|
||||
// the artifact import) and hold only credits naming more than one
|
||||
// artist: an entity with no rows here is credited to one artist, which
|
||||
// explore_index's own artist_name and artist_mbid already describe.
|
||||
// Absence is the common case and means "nothing to decompose", never
|
||||
// "unknown".
|
||||
//
|
||||
// The lookup is keyed on the *recording* MBID, which both sides of the
|
||||
// app already have -- a catalog row carries it and so does a local
|
||||
// file (library.Track.RecordingMBID) -- so one query serves the Explore
|
||||
// pages and the library's own lists without either needing to know
|
||||
// where the other gets its rows.
|
||||
|
||||
// CreditPart is one credited artist within a credit, in credit order.
|
||||
//
|
||||
// CreditedName is the name *as credited*, which is not the artist's own
|
||||
// name: MusicBrainz credits "Snoop Dogg" on a track by the artist
|
||||
// called "Snoop Doggy Dogg". Display uses it; navigation uses
|
||||
// ArtistMBID. JoinPhrase is the literal connector that follows this
|
||||
// part, so a credit renders by concatenation and never by searching a
|
||||
// name inside a credit string.
|
||||
type CreditPart struct {
|
||||
Position int `json:"position"`
|
||||
ArtistMBID string `json:"artistMbid"`
|
||||
CreditedName string `json:"creditedName"`
|
||||
JoinPhrase string `json:"joinPhrase"`
|
||||
}
|
||||
|
||||
// creditLookupBatch bounds how many MBIDs go into one IN clause. A
|
||||
// tracklist is the caller here, so the realistic ceiling is a few
|
||||
// hundred; the bound exists so a 50,000-row selection cannot build a
|
||||
// statement SQLite refuses to parse.
|
||||
const creditLookupBatch = 500
|
||||
|
||||
// GetCredits returns the decomposition of every multi-artist credit
|
||||
// among the given entity MBIDs, keyed by MBID.
|
||||
//
|
||||
// MBIDs with a single-artist credit are simply absent from the result,
|
||||
// which is what the caller wants: it renders its existing single link
|
||||
// for those, and that is the same answer it would have rendered anyway.
|
||||
func (si *SearchIndex) GetCredits(mbids []string) (map[string][]CreditPart, error) {
|
||||
out := make(map[string][]CreditPart)
|
||||
|
||||
for start := 0; start < len(mbids); start += creditLookupBatch {
|
||||
end := min(start+creditLookupBatch, len(mbids))
|
||||
|
||||
if err := si.appendCredits(mbids[start:end], out); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// appendCredits runs one batch into the accumulating result.
|
||||
func (si *SearchIndex) appendCredits(
|
||||
mbids []string, out map[string][]CreditPart,
|
||||
) error {
|
||||
args := make([]any, 0, len(mbids))
|
||||
holders := make([]string, 0, len(mbids))
|
||||
|
||||
for _, mbid := range mbids {
|
||||
if mbid == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
args = append(args, dbMBID(mbid))
|
||||
holders = append(holders, "?")
|
||||
}
|
||||
|
||||
if len(args) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Ordered by position because that ordering *is* the credit's
|
||||
// meaning; the caller concatenates in the order it receives.
|
||||
rows, err := si.db.QueryContext(
|
||||
`SELECT r.mbid, p.position, p.artist_mbid, p.credited_name, p.join_phrase
|
||||
FROM artist_credit_ref r
|
||||
JOIN artist_credit_part p ON p.credit_id = r.credit_id
|
||||
WHERE r.mbid IN (`+strings.Join(holders, ",")+`)
|
||||
ORDER BY r.mbid, p.position`,
|
||||
args...,
|
||||
)
|
||||
if err != nil {
|
||||
return fmt.Errorf("read artist credits: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
for rows.Next() {
|
||||
var (
|
||||
entity dbMBID
|
||||
artist dbMBID
|
||||
part CreditPart
|
||||
)
|
||||
|
||||
if err := rows.Scan(
|
||||
&entity, &part.Position, &artist, &part.CreditedName, &part.JoinPhrase,
|
||||
); err != nil {
|
||||
return fmt.Errorf("scan artist credit: %w", err)
|
||||
}
|
||||
|
||||
part.ArtistMBID = string(artist)
|
||||
out[string(entity)] = append(out[string(entity)], part)
|
||||
}
|
||||
|
||||
if err := rows.Err(); err != nil {
|
||||
return fmt.Errorf("read artist credits: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetCredits is the bound form: the frontend asks for a tracklist's
|
||||
// worth of MBIDs at once rather than one per row.
|
||||
//
|
||||
// Batched for the reason every other per-row backend question here is:
|
||||
// asking on hover or on render turns a list into N IPC round trips, and
|
||||
// this one is asked about every row of every list in the app.
|
||||
func (e *Service) GetCredits(mbids []string) (map[string][]CreditPart, error) {
|
||||
return e.index.GetCredits(mbids)
|
||||
}
|
||||
@@ -0,0 +1,128 @@
|
||||
package explore
|
||||
|
||||
import (
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"yellowjacket/backend/database"
|
||||
)
|
||||
|
||||
// seedCredit writes one multi-artist credit and points an entity at it,
|
||||
// the way the dump import and the artifact import both do.
|
||||
func seedCredit(t *testing.T, db *database.DB, entity string, id int, parts []CreditPart) {
|
||||
t.Helper()
|
||||
|
||||
pack := func(mbid string) []byte {
|
||||
raw, err := hex.DecodeString(strings.ReplaceAll(mbid, "-", ""))
|
||||
if err != nil || len(raw) != 16 {
|
||||
t.Fatalf("bad fixture mbid %q: %v", mbid, err)
|
||||
}
|
||||
|
||||
return raw
|
||||
}
|
||||
|
||||
if _, err := db.ExecContext(
|
||||
"INSERT INTO artist_credit_ref (mbid, credit_id) VALUES (?, ?)",
|
||||
pack(entity), id,
|
||||
); err != nil {
|
||||
t.Fatalf("seed ref: %v", err)
|
||||
}
|
||||
|
||||
for _, p := range parts {
|
||||
if _, err := db.ExecContext(
|
||||
`INSERT INTO artist_credit_part
|
||||
(credit_id, position, artist_mbid, credited_name, join_phrase)
|
||||
VALUES (?, ?, ?, ?, ?)`,
|
||||
id, p.Position, pack(p.ArtistMBID), p.CreditedName, p.JoinPhrase,
|
||||
); err != nil {
|
||||
t.Fatalf("seed part: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestGetCreditsDecomposes: the parts come back in position order and
|
||||
// concatenate to the credit they describe.
|
||||
func TestGetCreditsDecomposes(t *testing.T) {
|
||||
db := database.NewTestDB(t)
|
||||
si := NewSearchIndex(db, nil, nil, testLogger())
|
||||
|
||||
rec := testMBID("rec-1")
|
||||
a, b := testMBID("artist-a"), testMBID("artist-b")
|
||||
|
||||
seedCredit(t, db, rec, 7, []CreditPart{
|
||||
{Position: 0, ArtistMBID: a, CreditedName: "2Pac", JoinPhrase: " feat. "},
|
||||
{Position: 1, ArtistMBID: b, CreditedName: "Snoop Dogg"},
|
||||
})
|
||||
|
||||
got, err := si.GetCredits([]string{rec})
|
||||
if err != nil {
|
||||
t.Fatalf("GetCredits: %v", err)
|
||||
}
|
||||
|
||||
parts := got[rec]
|
||||
if len(parts) != 2 {
|
||||
t.Fatalf("parts = %d, want 2", len(parts))
|
||||
}
|
||||
|
||||
var rendered strings.Builder
|
||||
for _, p := range parts {
|
||||
rendered.WriteString(p.CreditedName)
|
||||
rendered.WriteString(p.JoinPhrase)
|
||||
}
|
||||
|
||||
if rendered.String() != "2Pac feat. Snoop Dogg" {
|
||||
t.Errorf("rendered = %q, want %q", rendered.String(), "2Pac feat. Snoop Dogg")
|
||||
}
|
||||
|
||||
// Dashed on the way out: a blob reaching the frontend is sixteen
|
||||
// bytes of mojibake, and nothing above mbid.go speaks that.
|
||||
if parts[0].ArtistMBID != a {
|
||||
t.Errorf("artist mbid = %q, want %q", parts[0].ArtistMBID, a)
|
||||
}
|
||||
}
|
||||
|
||||
// TestGetCreditsOmitsSingleArtist: absence is the common case and means
|
||||
// "nothing to decompose", so the caller renders its existing one link.
|
||||
func TestGetCreditsOmitsSingleArtist(t *testing.T) {
|
||||
db := database.NewTestDB(t)
|
||||
si := NewSearchIndex(db, nil, nil, testLogger())
|
||||
|
||||
got, err := si.GetCredits([]string{testMBID("untagged"), ""})
|
||||
if err != nil {
|
||||
t.Fatalf("GetCredits: %v", err)
|
||||
}
|
||||
|
||||
if len(got) != 0 {
|
||||
t.Errorf("got %d credits, want none", len(got))
|
||||
}
|
||||
}
|
||||
|
||||
// TestGetCreditsBatches: the lookup is asked about whole tracklists, so
|
||||
// it must not build one statement per row or one SQLite refuses to
|
||||
// parse.
|
||||
func TestGetCreditsBatches(t *testing.T) {
|
||||
db := database.NewTestDB(t)
|
||||
si := NewSearchIndex(db, nil, nil, testLogger())
|
||||
|
||||
mbids := make([]string, 0, creditLookupBatch*2+7)
|
||||
for i := range creditLookupBatch*2 + 7 {
|
||||
mbids = append(mbids, testMBID(fmt.Sprintf("batch-%d", i)))
|
||||
}
|
||||
|
||||
// One real credit somewhere past the first batch boundary.
|
||||
seedCredit(t, db, mbids[creditLookupBatch+3], 9, []CreditPart{
|
||||
{Position: 0, ArtistMBID: testMBID("a"), CreditedName: "A", JoinPhrase: " & "},
|
||||
{Position: 1, ArtistMBID: testMBID("b"), CreditedName: "B"},
|
||||
})
|
||||
|
||||
got, err := si.GetCredits(mbids)
|
||||
if err != nil {
|
||||
t.Fatalf("GetCredits: %v", err)
|
||||
}
|
||||
|
||||
if len(got[mbids[creditLookupBatch+3]]) != 2 {
|
||||
t.Errorf("a credit past the first batch boundary was not returned")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,637 @@
|
||||
//go:build indexbuild
|
||||
|
||||
package explore
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bufio"
|
||||
"compress/bzip2"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"path"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Multi-artist credits, from the core MusicBrainz dump.
|
||||
//
|
||||
// A credit is ordered parts and the credit *string* is derived from
|
||||
// them; MusicBrainz's own artist_credit.name is a cached render. What
|
||||
// this pass extracts is the decomposition: for each catalog recording
|
||||
// and release group whose credit names more than one artist, the
|
||||
// credited artists in order, each with the name *as credited* and the
|
||||
// join phrase that follows it. See artist_credit_part.sql for why that
|
||||
// is stored rather than derived, and why nothing may reconstruct a
|
||||
// credit by searching a name inside a credit string.
|
||||
//
|
||||
// It is a separate dump from everything else here, and it has to be.
|
||||
// The canonical dump this importer already streams gives artist_mbids
|
||||
// (an ordered list) and artist_credit_name (the *rendered* string) --
|
||||
// no join phrases, and no per-artist as-credited names. Splitting the
|
||||
// rendered string using canonical artist names fails on exactly the
|
||||
// credits that matter: measured on a real library, 21% of multi-artist
|
||||
// credits name an artist differently from the artist's own name
|
||||
// ("Snoop Dogg" credited on a track by "Snoop Doggy Dogg"), so the
|
||||
// substring is simply not there. The JSON dumps were checked too and
|
||||
// cover 153,691 recordings of ~35M, with zero overlap against a real
|
||||
// library. This dump is the only source.
|
||||
//
|
||||
// Cost, measured on the 20260815 export: 7.1 GB compressed, decompressed
|
||||
// by pure-Go compress/bzip2 at ~26 MB/s uncompressed (~13.7 min for the
|
||||
// whole file, single-threaded). cmd/indexbuild is built CGO_ENABLED=0,
|
||||
// so the stdlib decompressor is what there is -- and it is fine, because
|
||||
// the 2 MB/s origin throttle dominates, as it does for every other dump
|
||||
// here.
|
||||
|
||||
const (
|
||||
// defaultMBDumpBaseURL is the core MusicBrainz export. Only
|
||||
// mbdump.tar.bz2 is fetched; the other tarballs there hold data this
|
||||
// app has no use for.
|
||||
defaultMBDumpBaseURL = "https://data.metabrainz.org/pub/musicbrainz/data/fullexport/"
|
||||
)
|
||||
|
||||
var (
|
||||
mbdumpDirRe = regexp.MustCompile(`^\d{8}-\d+$`)
|
||||
mbdumpFileRe = regexp.MustCompile(`^mbdump\.tar\.bz2$`)
|
||||
|
||||
// ErrDumpShape is returned when a dump member does not have the
|
||||
// columns this code was written against. It is deliberately fatal:
|
||||
// reading the wrong column silently produces a catalog whose credits
|
||||
// are subtly wrong, which is far worse than a failed build.
|
||||
ErrDumpShape = errors.New("musicbrainz dump member has an unexpected shape")
|
||||
)
|
||||
|
||||
// Column positions in the Postgres COPY output, verified against the
|
||||
// 20260815 export. There is no header row to read them from, so they
|
||||
// are asserted instead -- see checkShape.
|
||||
const (
|
||||
artistColID = 0
|
||||
artistColGID = 1
|
||||
artistColMin = 2
|
||||
|
||||
creditColID = 0
|
||||
creditColArtistCount = 2
|
||||
creditColMin = 3
|
||||
|
||||
partColCredit = 0
|
||||
partColPosition = 1
|
||||
partColArtist = 2
|
||||
partColName = 3
|
||||
partColJoin = 4
|
||||
partColMin = 5
|
||||
|
||||
// recording and release_group share a layout in the columns this
|
||||
// pass reads: id, gid, name, artist_credit, ...
|
||||
entityColGID = 1
|
||||
entityColCredit = 3
|
||||
entityColMin = 4
|
||||
)
|
||||
|
||||
// creditPart is one credited artist within a credit.
|
||||
type creditPart struct {
|
||||
position int
|
||||
artistID int32
|
||||
name string
|
||||
join string
|
||||
}
|
||||
|
||||
// creditScan is what one pass over the dump collects.
|
||||
type creditScan struct {
|
||||
// artistGIDs maps an artist row id to its MBID. artist_credit_name
|
||||
// references artists by row id, and the tar orders `artist` before
|
||||
// it, so this is complete by the time it is read.
|
||||
artistGIDs map[int32]uuid16
|
||||
|
||||
// multiCredits are the credit ids naming more than one artist, from
|
||||
// artist_credit.artist_count. Taking the count from the dump rather
|
||||
// than counting parts means a credit can be rejected before its
|
||||
// parts are stored.
|
||||
multiCredits map[int32]struct{}
|
||||
|
||||
// parts are the decompositions of multiCredits, keyed by credit id.
|
||||
parts map[int32][]creditPart
|
||||
|
||||
// refs maps a kept catalog entity to its credit. Only entities in
|
||||
// explore_index and only multi-artist credits: everything else is
|
||||
// already described by explore_index's own artist_name/artist_mbid.
|
||||
refs map[uuid16]int32
|
||||
|
||||
// used are the credits some ref actually points at, which is a small
|
||||
// fraction of multiCredits -- the catalog keeps ~1.8M entities of
|
||||
// MusicBrainz's tens of millions.
|
||||
used map[int32]struct{}
|
||||
|
||||
skippedUnknownArtist int
|
||||
}
|
||||
|
||||
// creditsImportDoneKey marks in explore_index_meta that the credit pass
|
||||
// has run against the current catalog.
|
||||
//
|
||||
// It is its own marker rather than part of the import's stage state for
|
||||
// a resume reason: the credit pass runs *after* the catalog is
|
||||
// assembled, and a failure in it must not send the next run back
|
||||
// through the ~205 GB it just finished. Marking separately means a
|
||||
// retry retries only this.
|
||||
const creditsImportDoneKey = "credits_import_done"
|
||||
|
||||
// ensureArtistCredits runs the credit pass unless it has already run
|
||||
// against this catalog, reporting whether it newly populated them.
|
||||
//
|
||||
// Called from both of run's paths -- the full import and the resume
|
||||
// that finds the rows already assembled -- and from the maintenance
|
||||
// entry point below, since a catalog built before credits existed is
|
||||
// otherwise never offered a chance to gain them: the index job picks
|
||||
// its mode from the index's own state, and a complete import means
|
||||
// "refresh", which never enters run() at all.
|
||||
//
|
||||
// The return value is what tells the job there is something new worth
|
||||
// publishing. A refresh otherwise reports "changed" only when the
|
||||
// listens series advanced, so credits would sit in the CI database and
|
||||
// never reach an artifact.
|
||||
func (imp *dumpImporter) ensureArtistCredits(ctx context.Context) bool {
|
||||
if imp.si.hasMeta(creditsImportDoneKey) {
|
||||
return false
|
||||
}
|
||||
|
||||
url, err := discoverDumpFile(
|
||||
ctx, imp.httpClient, imp.mbdumpBaseURL, mbdumpDirRe, mbdumpFileRe,
|
||||
)
|
||||
if err != nil {
|
||||
imp.logger.Warn("credit import: could not find the dump", "error", err)
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
if err := imp.importArtistCredits(ctx, url); err != nil {
|
||||
// A catalog without credits is the catalog this app shipped
|
||||
// before them: every credit falls back to its single artist.
|
||||
// That is worth far less than failing an import that otherwise
|
||||
// succeeded.
|
||||
imp.logger.Warn("credit import: failed", "error", err)
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
imp.si.setMeta(creditsImportDoneKey, "1")
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
// EnsureArtistCredits tops up the credit tables outside a full import.
|
||||
//
|
||||
// It exists because the index job's modes are decided from the index's
|
||||
// own state: a cache holding a completed import chooses `refresh`,
|
||||
// which folds in incremental listens and never enters the dump
|
||||
// importer. Without this, a catalog built before the credit pass
|
||||
// existed could only gain credits from a `rebuild` -- and a rebuild
|
||||
// re-downloads ~205 GB to reproduce rows it already has, to add
|
||||
// something that costs 7 GB on its own.
|
||||
//
|
||||
// Reports whether credits were newly populated, so the caller knows
|
||||
// there is a new artifact worth publishing.
|
||||
func (e *Service) EnsureArtistCredits(ctx context.Context) bool {
|
||||
imp, err := newDumpImporter(e.index, e.lb)
|
||||
if err != nil {
|
||||
e.index.logger.Warn("credit import: could not start", "error", err)
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
return imp.ensureArtistCredits(ctx)
|
||||
}
|
||||
|
||||
// importArtistCredits streams the core MusicBrainz dump and fills
|
||||
// artist_credit_part and artist_credit_ref for the entities the catalog
|
||||
// kept.
|
||||
//
|
||||
// It runs after assembleIndex because it asks explore_index which
|
||||
// entities those are: the popularity filter decides what is worth
|
||||
// carrying credits for, and asking the table rather than the kept sets
|
||||
// means this stays correct if that filter changes.
|
||||
func (imp *dumpImporter) importArtistCredits(ctx context.Context, url string) error {
|
||||
kept, err := imp.keptEntityMBIDs(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if len(kept) == 0 {
|
||||
imp.logger.Warn("credit import: no catalog entities, skipping")
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
imp.logger.Info("credit import: starting", "url", url, "entities", len(kept))
|
||||
imp.logJob("Streaming MusicBrainz dump for artist credits")
|
||||
|
||||
scan, err := imp.scanCreditDump(ctx, url, kept)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
imp.logger.Info("credit import: scanned",
|
||||
"multiArtistCredits", len(scan.multiCredits),
|
||||
"entitiesWithMultiArtistCredit", len(scan.refs),
|
||||
"creditsUsed", len(scan.used),
|
||||
)
|
||||
|
||||
return imp.writeCredits(ctx, scan)
|
||||
}
|
||||
|
||||
// keptEntityMBIDs is every recording and release group in the catalog.
|
||||
// Artists are excluded: an artist is not credited to a credit.
|
||||
func (imp *dumpImporter) keptEntityMBIDs(ctx context.Context) (map[uuid16]struct{}, error) {
|
||||
rows, err := imp.si.db.QueryContextWith(ctx,
|
||||
`SELECT mbid FROM explore_index
|
||||
WHERE entity_type IN (2 /* release_group */, 3 /* recording */)`,
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("credit import: read catalog entities: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
out := make(map[uuid16]struct{})
|
||||
|
||||
for rows.Next() {
|
||||
var raw []byte
|
||||
|
||||
if err := rows.Scan(&raw); err != nil {
|
||||
return nil, fmt.Errorf("credit import: scan mbid: %w", err)
|
||||
}
|
||||
|
||||
if len(raw) != len(uuid16{}) {
|
||||
continue
|
||||
}
|
||||
|
||||
var id uuid16
|
||||
|
||||
copy(id[:], raw)
|
||||
|
||||
out[id] = struct{}{}
|
||||
}
|
||||
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, fmt.Errorf("credit import: read catalog entities: %w", err)
|
||||
}
|
||||
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// scanCreditDump makes one sequential pass over mbdump.tar.bz2.
|
||||
//
|
||||
// The tar's members are alphabetical, which is what makes a single pass
|
||||
// possible without buffering the big ones: `artist` and
|
||||
// `artist_credit_name` both arrive before `recording` and
|
||||
// `release_group`, so by the time an entity names a credit, that
|
||||
// credit's parts and their artists' MBIDs are already known and the
|
||||
// entity can be resolved and dropped. 35M recording rows are never
|
||||
// held.
|
||||
//
|
||||
// The order is not depended on blindly: an entity naming a credit that
|
||||
// has not been seen is counted and reported rather than silently
|
||||
// producing an empty catalog, which is what a reordered export would
|
||||
// otherwise look like.
|
||||
func (imp *dumpImporter) scanCreditDump(
|
||||
ctx context.Context, url string, kept map[uuid16]struct{},
|
||||
) (*creditScan, error) {
|
||||
stream := imp.openDumpStream(ctx, url, 0)
|
||||
|
||||
defer func() { _ = stream.Close() }()
|
||||
|
||||
return imp.scanCreditTar(
|
||||
ctx,
|
||||
tar.NewReader(bzip2.NewReader(bufio.NewReaderSize(stream, 1<<20))),
|
||||
kept,
|
||||
)
|
||||
}
|
||||
|
||||
// scanCreditTar is the parse, separated from the fetch so it can be
|
||||
// driven by a tar built in a test. compress/bzip2 is decompress-only,
|
||||
// so a test cannot produce the real container.
|
||||
func (imp *dumpImporter) scanCreditTar(
|
||||
ctx context.Context, tr *tar.Reader, kept map[uuid16]struct{},
|
||||
) (*creditScan, error) {
|
||||
scan := &creditScan{
|
||||
artistGIDs: make(map[int32]uuid16),
|
||||
multiCredits: make(map[int32]struct{}),
|
||||
parts: make(map[int32][]creditPart),
|
||||
refs: make(map[uuid16]int32),
|
||||
used: make(map[int32]struct{}),
|
||||
}
|
||||
|
||||
for {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
hdr, err := tr.Next()
|
||||
if errors.Is(err, io.EOF) {
|
||||
break
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("credit import: tar: %w", err)
|
||||
}
|
||||
|
||||
if hdr.Typeflag != tar.TypeReg {
|
||||
continue
|
||||
}
|
||||
|
||||
done, err := imp.scanCreditMember(ctx, hdr.Name, tr, kept, scan)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if done {
|
||||
// Everything this pass needs has been read; the rest of the
|
||||
// tarball is other entities' data and decompressing it would
|
||||
// cost minutes for nothing.
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
if scan.skippedUnknownArtist > 0 {
|
||||
imp.logger.Warn("credit import: credits dropped for unknown artists",
|
||||
"count", scan.skippedUnknownArtist,
|
||||
)
|
||||
}
|
||||
|
||||
return scan, nil
|
||||
}
|
||||
|
||||
// scanCreditMember dispatches one tar member, reporting whether the
|
||||
// pass has everything it needs.
|
||||
func (imp *dumpImporter) scanCreditMember(
|
||||
ctx context.Context, name string, r io.Reader,
|
||||
kept map[uuid16]struct{}, scan *creditScan,
|
||||
) (bool, error) {
|
||||
switch path.Base(name) {
|
||||
case "artist":
|
||||
return false, imp.scanArtists(ctx, r, scan)
|
||||
case "artist_credit":
|
||||
return false, imp.scanCredits(ctx, r, scan)
|
||||
case "artist_credit_name":
|
||||
return false, imp.scanCreditParts(ctx, r, scan)
|
||||
case "recording", "release_group":
|
||||
if err := imp.scanCreditedEntities(ctx, r, kept, scan); err != nil {
|
||||
return false, err
|
||||
}
|
||||
|
||||
// release_group sorts after recording, so the pass is complete
|
||||
// once it has been read.
|
||||
return path.Base(name) == "release_group", nil
|
||||
default:
|
||||
return false, nil
|
||||
}
|
||||
}
|
||||
|
||||
// scanArtists records every artist's MBID by row id.
|
||||
func (imp *dumpImporter) scanArtists(
|
||||
ctx context.Context, r io.Reader, scan *creditScan,
|
||||
) error {
|
||||
return scanTSV(ctx, r, artistColMin, "artist", func(fields []string) error {
|
||||
id, ok := parseInt32(fields[artistColID])
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
|
||||
var gid uuid16
|
||||
|
||||
if !parseUUID(fields[artistColGID], gid[:]) {
|
||||
return fmt.Errorf("%w: artist.gid is not a UUID: %q",
|
||||
ErrDumpShape, truncate(fields[artistColGID]))
|
||||
}
|
||||
|
||||
scan.artistGIDs[id] = gid
|
||||
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// scanCredits records which credits name more than one artist.
|
||||
func (imp *dumpImporter) scanCredits(
|
||||
ctx context.Context, r io.Reader, scan *creditScan,
|
||||
) error {
|
||||
return scanTSV(ctx, r, creditColMin, "artist_credit", func(fields []string) error {
|
||||
id, ok := parseInt32(fields[creditColID])
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
|
||||
count, ok := parseInt32(fields[creditColArtistCount])
|
||||
if !ok {
|
||||
return fmt.Errorf("%w: artist_credit.artist_count is not a number: %q",
|
||||
ErrDumpShape, truncate(fields[creditColArtistCount]))
|
||||
}
|
||||
|
||||
if count > 1 {
|
||||
scan.multiCredits[id] = struct{}{}
|
||||
}
|
||||
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// scanCreditParts records the decomposition of every multi-artist
|
||||
// credit.
|
||||
func (imp *dumpImporter) scanCreditParts(
|
||||
ctx context.Context, r io.Reader, scan *creditScan,
|
||||
) error {
|
||||
return scanTSV(ctx, r, partColMin, "artist_credit_name", func(fields []string) error {
|
||||
credit, ok := parseInt32(fields[partColCredit])
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
|
||||
if _, multi := scan.multiCredits[credit]; !multi {
|
||||
return nil
|
||||
}
|
||||
|
||||
position, ok := parseInt32(fields[partColPosition])
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
|
||||
artist, ok := parseInt32(fields[partColArtist])
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
|
||||
scan.parts[credit] = append(scan.parts[credit], creditPart{
|
||||
position: int(position),
|
||||
artistID: artist,
|
||||
name: fields[partColName],
|
||||
join: fields[partColJoin],
|
||||
})
|
||||
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// scanCreditedEntities resolves recordings and release groups against
|
||||
// the catalog, keeping only those the catalog holds and whose credit
|
||||
// names more than one artist.
|
||||
func (imp *dumpImporter) scanCreditedEntities(
|
||||
ctx context.Context, r io.Reader, kept map[uuid16]struct{}, scan *creditScan,
|
||||
) error {
|
||||
return scanTSV(ctx, r, entityColMin, "recording/release_group",
|
||||
func(fields []string) error {
|
||||
var gid uuid16
|
||||
|
||||
if !parseUUID(fields[entityColGID], gid[:]) {
|
||||
return fmt.Errorf("%w: entity gid is not a UUID: %q",
|
||||
ErrDumpShape, truncate(fields[entityColGID]))
|
||||
}
|
||||
|
||||
if _, want := kept[gid]; !want {
|
||||
return nil
|
||||
}
|
||||
|
||||
credit, ok := parseInt32(fields[entityColCredit])
|
||||
if !ok {
|
||||
return fmt.Errorf("%w: entity artist_credit is not a number: %q",
|
||||
ErrDumpShape, truncate(fields[entityColCredit]))
|
||||
}
|
||||
|
||||
if _, multi := scan.multiCredits[credit]; !multi {
|
||||
return nil
|
||||
}
|
||||
|
||||
scan.refs[gid] = credit
|
||||
scan.used[credit] = struct{}{}
|
||||
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// scanTSV reads Postgres COPY output a line at a time, unescaping each
|
||||
// field and handing the row to fn.
|
||||
//
|
||||
// The shape is asserted on the first row rather than trusted: this dump
|
||||
// has no header, so a column that moved would otherwise be read as a
|
||||
// neighbouring one and produce a catalog that is quietly wrong.
|
||||
func scanTSV(
|
||||
ctx context.Context, r io.Reader, minCols int, member string,
|
||||
fn func(fields []string) error,
|
||||
) error {
|
||||
sc := bufio.NewScanner(r)
|
||||
sc.Buffer(make([]byte, 0, 1<<20), 1<<24)
|
||||
|
||||
checked := false
|
||||
rows := 0
|
||||
|
||||
for sc.Scan() {
|
||||
rows++
|
||||
|
||||
if rows%(1<<20) == 0 {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
line := sc.Text()
|
||||
if line == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
fields := strings.Split(line, "\t")
|
||||
if len(fields) < minCols {
|
||||
if !checked {
|
||||
return fmt.Errorf("%w: %s has %d columns, need at least %d",
|
||||
ErrDumpShape, member, len(fields), minCols)
|
||||
}
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
checked = true
|
||||
|
||||
for i := range fields {
|
||||
fields[i] = unescapeCopy(fields[i])
|
||||
}
|
||||
|
||||
if err := fn(fields); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
if err := sc.Err(); err != nil {
|
||||
return fmt.Errorf("credit import: read %s: %w", member, err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// unescapeCopy undoes Postgres COPY's text escaping. A NULL (\N) is
|
||||
// returned as an empty string: every field this pass reads is either a
|
||||
// number it will reject or a name whose absence means the same as
|
||||
// empty.
|
||||
func unescapeCopy(s string) string {
|
||||
if s == `\N` {
|
||||
return ""
|
||||
}
|
||||
|
||||
if !strings.ContainsRune(s, '\\') {
|
||||
return s
|
||||
}
|
||||
|
||||
var b strings.Builder
|
||||
|
||||
b.Grow(len(s))
|
||||
|
||||
for i := 0; i < len(s); i++ {
|
||||
if s[i] != '\\' || i+1 >= len(s) {
|
||||
b.WriteByte(s[i])
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
i++
|
||||
|
||||
switch s[i] {
|
||||
case 'n':
|
||||
b.WriteByte('\n')
|
||||
case 't':
|
||||
b.WriteByte('\t')
|
||||
case 'r':
|
||||
b.WriteByte('\r')
|
||||
case 'b':
|
||||
b.WriteByte('\b')
|
||||
case 'f':
|
||||
b.WriteByte('\f')
|
||||
case 'v':
|
||||
b.WriteByte('\v')
|
||||
case '\\':
|
||||
b.WriteByte('\\')
|
||||
default:
|
||||
b.WriteByte('\\')
|
||||
b.WriteByte(s[i])
|
||||
}
|
||||
}
|
||||
|
||||
return b.String()
|
||||
}
|
||||
|
||||
func parseInt32(s string) (int32, bool) {
|
||||
n, err := strconv.ParseInt(s, 10, 32)
|
||||
if err != nil {
|
||||
return 0, false
|
||||
}
|
||||
|
||||
return int32(n), true
|
||||
}
|
||||
|
||||
// truncate bounds an error message built from dump data, which is
|
||||
// attacker-free but can be long.
|
||||
func truncate(s string) string {
|
||||
const limit = 64
|
||||
|
||||
if len(s) <= limit {
|
||||
return s
|
||||
}
|
||||
|
||||
return s[:limit] + "..."
|
||||
}
|
||||
@@ -0,0 +1,406 @@
|
||||
//go:build indexbuild
|
||||
|
||||
package explore
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"yellowjacket/backend/database"
|
||||
)
|
||||
|
||||
// tarOf builds an uncompressed tar of the named members, in the order
|
||||
// given. Order is the point of several of these tests: the real dump's
|
||||
// members are alphabetical, which is what lets one pass resolve an
|
||||
// entity's credit without buffering 35M recordings.
|
||||
func tarOf(t *testing.T, members ...[2]string) *tar.Reader {
|
||||
t.Helper()
|
||||
|
||||
var buf bytes.Buffer
|
||||
|
||||
tw := tar.NewWriter(&buf)
|
||||
|
||||
for _, m := range members {
|
||||
body := []byte(m[1])
|
||||
|
||||
if err := tw.WriteHeader(&tar.Header{
|
||||
Name: "mbdump/" + m[0],
|
||||
Mode: 0o644,
|
||||
Size: int64(len(body)),
|
||||
Typeflag: tar.TypeReg,
|
||||
}); err != nil {
|
||||
t.Fatalf("tar header: %v", err)
|
||||
}
|
||||
|
||||
if _, err := tw.Write(body); err != nil {
|
||||
t.Fatalf("tar write: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
if err := tw.Close(); err != nil {
|
||||
t.Fatalf("tar close: %v", err)
|
||||
}
|
||||
|
||||
return tar.NewReader(&buf)
|
||||
}
|
||||
|
||||
func tsv(rows ...[]string) string {
|
||||
var b strings.Builder
|
||||
|
||||
for _, r := range rows {
|
||||
b.WriteString(strings.Join(r, "\t"))
|
||||
b.WriteByte('\n')
|
||||
}
|
||||
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// mustMBID is testMBID in the packed form the catalog stores.
|
||||
func mustMBID(label string) uuid16 {
|
||||
var u uuid16
|
||||
|
||||
if !parseUUID(testMBID(label), u[:]) {
|
||||
panic("testMBID did not produce a UUID for " + label)
|
||||
}
|
||||
|
||||
return u
|
||||
}
|
||||
|
||||
// The two artists of the worked example, and the entities they credit.
|
||||
var (
|
||||
creditRecMBID = mustMBID("recording-1")
|
||||
creditRGMBID = mustMBID("release-group-1")
|
||||
)
|
||||
|
||||
// sampleDump is the shape verified against the 20260815 export:
|
||||
// artist(id, gid, ...), artist_credit(id, name, artist_count, ...),
|
||||
// artist_credit_name(credit, position, artist, name, join_phrase),
|
||||
// recording/release_group(id, gid, name, artist_credit, ...).
|
||||
func sampleDump(t *testing.T) *tar.Reader {
|
||||
t.Helper()
|
||||
|
||||
return tarOf(t,
|
||||
[2]string{"artist", tsv(
|
||||
[]string{"11", testMBID("artist-a"), "Snoop Doggy Dogg", "Snoop Doggy Dogg"},
|
||||
[]string{"22", testMBID("artist-b"), "2Pac", "2Pac"},
|
||||
)},
|
||||
[2]string{"artist_credit", tsv(
|
||||
[]string{"900", "2Pac feat. Snoop Dogg", "2", "1", "", "0", ""},
|
||||
[]string{"901", "Solo Artist", "1", "1", "", "0", ""},
|
||||
)},
|
||||
[2]string{"artist_credit_name", tsv(
|
||||
// Deliberately out of position order: the dump is not
|
||||
// obliged to emit them sorted and the credit's meaning is
|
||||
// the order, not the file's.
|
||||
[]string{"900", "1", "11", "Snoop Dogg", ""},
|
||||
[]string{"900", "0", "22", "2Pac", " feat. "},
|
||||
[]string{"901", "0", "11", "Solo Artist", ""},
|
||||
)},
|
||||
[2]string{"recording", tsv(
|
||||
[]string{"1", testMBID("recording-1"), "Some Song", "900", "180000"},
|
||||
[]string{"2", testMBID("not-kept"), "Other", "900", "1"},
|
||||
[]string{"3", testMBID("solo"), "Solo", "901", "1"},
|
||||
)},
|
||||
[2]string{"release_group", tsv(
|
||||
[]string{"5", testMBID("release-group-1"), "Some Album", "900", "1"},
|
||||
)},
|
||||
)
|
||||
}
|
||||
|
||||
func creditTestImporter(t *testing.T) *dumpImporter {
|
||||
t.Helper()
|
||||
|
||||
db := database.NewTestDB(t)
|
||||
|
||||
return &dumpImporter{
|
||||
si: NewSearchIndex(db, nil, nil, testLogger()),
|
||||
logger: testLogger(),
|
||||
}
|
||||
}
|
||||
|
||||
// TestScanCreditDumpDecomposes is the worked example end to end: the
|
||||
// credit's parts come back in position order, with the *credited*
|
||||
// names and the join phrase between them.
|
||||
func TestScanCreditDumpDecomposes(t *testing.T) {
|
||||
imp := creditTestImporter(t)
|
||||
|
||||
kept := map[uuid16]struct{}{
|
||||
creditRecMBID: {},
|
||||
creditRGMBID: {},
|
||||
}
|
||||
|
||||
scan, err := imp.scanCreditTar(context.Background(), sampleDump(t), kept)
|
||||
if err != nil {
|
||||
t.Fatalf("scan: %v", err)
|
||||
}
|
||||
|
||||
if got := len(scan.refs); got != 2 {
|
||||
t.Fatalf("refs = %d, want 2 (the recording and the release group)", got)
|
||||
}
|
||||
|
||||
if scan.refs[creditRecMBID] != 900 {
|
||||
t.Errorf("recording credit = %d, want 900", scan.refs[creditRecMBID])
|
||||
}
|
||||
|
||||
parts := scan.parts[900]
|
||||
if len(parts) != 2 {
|
||||
t.Fatalf("parts = %d, want 2", len(parts))
|
||||
}
|
||||
|
||||
// Sorting happens on write, so assert the pieces are all present
|
||||
// and let the render test below check the order.
|
||||
byPos := map[int]creditPart{}
|
||||
for _, p := range parts {
|
||||
byPos[p.position] = p
|
||||
}
|
||||
|
||||
if byPos[0].name != "2Pac" || byPos[0].join != " feat. " {
|
||||
t.Errorf("position 0 = %q/%q, want \"2Pac\"/\" feat. \"",
|
||||
byPos[0].name, byPos[0].join)
|
||||
}
|
||||
|
||||
// The credited name, not the artist's own name: this is the whole
|
||||
// reason credited_name is stored per row.
|
||||
if byPos[1].name != "Snoop Dogg" {
|
||||
t.Errorf("position 1 credited name = %q, want \"Snoop Dogg\"", byPos[1].name)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSingleArtistCreditsAreNotStored: a one-artist credit is already
|
||||
// described by explore_index's artist_name/artist_mbid, and storing it
|
||||
// would roughly triple the table to say nothing new.
|
||||
func TestSingleArtistCreditsAreNotStored(t *testing.T) {
|
||||
imp := creditTestImporter(t)
|
||||
|
||||
solo := mustMBID("solo")
|
||||
kept := map[uuid16]struct{}{solo: {}}
|
||||
|
||||
scan, err := imp.scanCreditTar(context.Background(), sampleDump(t), kept)
|
||||
if err != nil {
|
||||
t.Fatalf("scan: %v", err)
|
||||
}
|
||||
|
||||
if len(scan.refs) != 0 {
|
||||
t.Fatalf("a single-artist credit was referenced: %v", scan.refs)
|
||||
}
|
||||
|
||||
if _, ok := scan.multiCredits[901]; ok {
|
||||
t.Error("credit 901 has artist_count 1 and should not be multi")
|
||||
}
|
||||
}
|
||||
|
||||
// TestOnlyKeptEntitiesAreReferenced: the catalog's popularity filter
|
||||
// decides what is worth carrying credits for, and an entity outside it
|
||||
// must not produce a row pointing at nothing.
|
||||
func TestOnlyKeptEntitiesAreReferenced(t *testing.T) {
|
||||
imp := creditTestImporter(t)
|
||||
|
||||
kept := map[uuid16]struct{}{creditRecMBID: {}}
|
||||
|
||||
scan, err := imp.scanCreditTar(context.Background(), sampleDump(t), kept)
|
||||
if err != nil {
|
||||
t.Fatalf("scan: %v", err)
|
||||
}
|
||||
|
||||
if _, ok := scan.refs[mustMBID("not-kept")]; ok {
|
||||
t.Error("an entity outside the catalog was referenced")
|
||||
}
|
||||
|
||||
if len(scan.used) != 1 {
|
||||
t.Errorf("used credits = %d, want 1", len(scan.used))
|
||||
}
|
||||
}
|
||||
|
||||
// TestWriteCreditsRoundTrips checks what the frontend will actually
|
||||
// read: parts in position order, dashed MBIDs out of the 16 raw bytes,
|
||||
// and a rendered credit that reassembles to the tagged string.
|
||||
func TestWriteCreditsRoundTrips(t *testing.T) {
|
||||
imp := creditTestImporter(t)
|
||||
|
||||
kept := map[uuid16]struct{}{creditRecMBID: {}, creditRGMBID: {}}
|
||||
|
||||
scan, err := imp.scanCreditTar(context.Background(), sampleDump(t), kept)
|
||||
if err != nil {
|
||||
t.Fatalf("scan: %v", err)
|
||||
}
|
||||
|
||||
if err := imp.writeCredits(context.Background(), scan); err != nil {
|
||||
t.Fatalf("writeCredits: %v", err)
|
||||
}
|
||||
|
||||
rows, err := imp.si.db.QueryContext(
|
||||
`SELECT p.position, p.artist_mbid, p.credited_name, p.join_phrase
|
||||
FROM artist_credit_ref r
|
||||
JOIN artist_credit_part p ON p.credit_id = r.credit_id
|
||||
WHERE r.mbid = ?
|
||||
ORDER BY p.position`,
|
||||
creditRecMBID[:],
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("query: %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
var rendered strings.Builder
|
||||
|
||||
names := []string{}
|
||||
|
||||
for rows.Next() {
|
||||
var (
|
||||
pos int
|
||||
mbid []byte
|
||||
name string
|
||||
join string
|
||||
)
|
||||
|
||||
if err := rows.Scan(&pos, &mbid, &name, &join); err != nil {
|
||||
t.Fatalf("scan row: %v", err)
|
||||
}
|
||||
|
||||
if len(mbid) != 16 {
|
||||
t.Fatalf("artist_mbid is %d bytes, want 16", len(mbid))
|
||||
}
|
||||
|
||||
names = append(names, name)
|
||||
|
||||
rendered.WriteString(name)
|
||||
rendered.WriteString(join)
|
||||
}
|
||||
|
||||
if err := rows.Err(); err != nil {
|
||||
t.Fatalf("rows: %v", err)
|
||||
}
|
||||
|
||||
// Concatenation is the contract: names in order, join phrases
|
||||
// between them, and no searching a name inside a credit string.
|
||||
if got := rendered.String(); got != "2Pac feat. Snoop Dogg" {
|
||||
t.Errorf("rendered credit = %q, want %q", got, "2Pac feat. Snoop Dogg")
|
||||
}
|
||||
|
||||
if len(names) != 2 || names[0] != "2Pac" {
|
||||
t.Errorf("parts came back out of position order: %v", names)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCreditRefsNeverDangle: a ref whose parts were not stored renders
|
||||
// as a credit with no artists at all, which is worse than the
|
||||
// single-artist fallback it replaced.
|
||||
func TestCreditRefsNeverDangle(t *testing.T) {
|
||||
imp := creditTestImporter(t)
|
||||
|
||||
kept := map[uuid16]struct{}{creditRecMBID: {}}
|
||||
|
||||
scan, err := imp.scanCreditTar(context.Background(), sampleDump(t), kept)
|
||||
if err != nil {
|
||||
t.Fatalf("scan: %v", err)
|
||||
}
|
||||
|
||||
// An artist the dump never named: the credit cannot be navigated to
|
||||
// and must be dropped whole, taking its ref with it.
|
||||
scan.artistGIDs = map[int32]uuid16{}
|
||||
|
||||
if err := imp.writeCredits(context.Background(), scan); err != nil {
|
||||
t.Fatalf("writeCredits: %v", err)
|
||||
}
|
||||
|
||||
var refs, parts int
|
||||
|
||||
if err := imp.si.db.QueryRowWriter(
|
||||
"SELECT COUNT(*) FROM artist_credit_ref",
|
||||
).Scan(&refs); err != nil {
|
||||
t.Fatalf("count refs: %v", err)
|
||||
}
|
||||
|
||||
if err := imp.si.db.QueryRowWriter(
|
||||
"SELECT COUNT(*) FROM artist_credit_part",
|
||||
).Scan(&parts); err != nil {
|
||||
t.Fatalf("count parts: %v", err)
|
||||
}
|
||||
|
||||
if refs != 0 || parts != 0 {
|
||||
t.Fatalf("refs=%d parts=%d, want 0/0 when the artists are unknown", refs, parts)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCreditDumpShapeIsAsserted: the dump has no header row, so a
|
||||
// column that moved would be read as its neighbour and produce a
|
||||
// catalog that is quietly wrong. Loud is the requirement.
|
||||
func TestCreditDumpShapeIsAsserted(t *testing.T) {
|
||||
imp := creditTestImporter(t)
|
||||
|
||||
short := tarOf(t, [2]string{"artist", tsv([]string{"11", "only-two-columns"})})
|
||||
|
||||
_, err := imp.scanCreditTar(context.Background(), short, map[uuid16]struct{}{})
|
||||
if err == nil {
|
||||
t.Fatal("a member with a non-UUID gid was accepted")
|
||||
}
|
||||
|
||||
if !errors.Is(err, ErrDumpShape) {
|
||||
t.Errorf("error = %v, want ErrDumpShape", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestUnescapeCopy covers Postgres COPY's text escaping, which reaches
|
||||
// artist names routinely -- a tab or backslash in a name would
|
||||
// otherwise shift every field after it.
|
||||
func TestUnescapeCopy(t *testing.T) {
|
||||
tests := []struct{ in, want string }{
|
||||
{`plain`, `plain`},
|
||||
{`\N`, ``},
|
||||
{`a\tb`, "a\tb"},
|
||||
{`a\nb`, "a\nb"},
|
||||
{`back\\slash`, `back\slash`},
|
||||
{`AC\/DC`, `AC\/DC`},
|
||||
{`trailing\`, `trailing\`},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
if got := unescapeCopy(tt.in); got != tt.want {
|
||||
t.Errorf("unescapeCopy(%q) = %q, want %q", tt.in, got, tt.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestEnsureArtistCreditsIsIdempotent pins what the index job depends
|
||||
// on to decide whether to publish.
|
||||
//
|
||||
// The pass runs on every mode, including the `refresh` that a complete
|
||||
// catalog always chooses — so it must be free when there is nothing to
|
||||
// do, and it must say so. A `true` here republishes the artifact; a
|
||||
// `true` on every run would republish an identical one weekly, and a
|
||||
// permanent `false` would mean a catalog that never gains credits at
|
||||
// all.
|
||||
func TestEnsureArtistCreditsIsIdempotent(t *testing.T) {
|
||||
imp := creditTestImporter(t)
|
||||
|
||||
// The marker is what "already done" means; with it set, the pass
|
||||
// must not reach the network or report a change.
|
||||
imp.si.setMeta(creditsImportDoneKey, "1")
|
||||
|
||||
if imp.ensureArtistCredits(context.Background()) {
|
||||
t.Fatal("a second run reported new credits; the artifact would republish forever")
|
||||
}
|
||||
}
|
||||
|
||||
// TestEnsureArtistCreditsReportsFailureAsNoChange: a dump that cannot be
|
||||
// reached leaves the catalog exactly as it was, and must not claim
|
||||
// otherwise — publishing on it would ship an artifact with no credits
|
||||
// and mark the work done.
|
||||
func TestEnsureArtistCreditsReportsFailureAsNoChange(t *testing.T) {
|
||||
imp := creditTestImporter(t)
|
||||
imp.httpClient = newDumpHTTPClient()
|
||||
imp.mbdumpBaseURL = "http://127.0.0.1:1/nonexistent/"
|
||||
|
||||
if imp.ensureArtistCredits(context.Background()) {
|
||||
t.Fatal("an unreachable dump reported new credits")
|
||||
}
|
||||
|
||||
if imp.si.hasMeta(creditsImportDoneKey) {
|
||||
t.Error("a failed pass marked itself done; it would never retry")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,164 @@
|
||||
//go:build indexbuild
|
||||
|
||||
package explore
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"sort"
|
||||
)
|
||||
|
||||
// writeCredits persists the scanned decompositions.
|
||||
//
|
||||
// Only credits some catalog entity actually points at are written: the
|
||||
// dump has millions of multi-artist credits and the catalog keeps ~1.8M
|
||||
// entities, so storing every credit would be most of a table nothing
|
||||
// can reach.
|
||||
//
|
||||
// The two tables are written in one transaction, because a ref pointing
|
||||
// at parts that are not there renders as a credit with no artists --
|
||||
// worse than the single-artist fallback it replaced.
|
||||
func (imp *dumpImporter) writeCredits(ctx context.Context, scan *creditScan) error {
|
||||
tx, err := imp.si.db.BeginTx()
|
||||
if err != nil {
|
||||
return fmt.Errorf("credit import: begin: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = tx.Rollback() }()
|
||||
|
||||
// A rebuild replaces the previous pass wholesale. These are Cache
|
||||
// tables derived entirely from the dump, so there is nothing to
|
||||
// merge and a stale row is a wrong credit.
|
||||
for _, table := range []string{"artist_credit_part", "artist_credit_ref"} {
|
||||
if _, err := tx.ExecContext(ctx, "DELETE FROM "+table); err != nil {
|
||||
return fmt.Errorf("credit import: clear %s: %w", table, err)
|
||||
}
|
||||
}
|
||||
|
||||
written, err := imp.writeCreditParts(ctx, tx, scan)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
refs, err := imp.writeCreditRefs(ctx, tx, scan, written)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if err := tx.Commit(); err != nil {
|
||||
return fmt.Errorf("credit import: commit: %w", err)
|
||||
}
|
||||
|
||||
imp.logger.Info("credit import: complete",
|
||||
"credits", len(written),
|
||||
"refs", refs,
|
||||
)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// writeCreditParts inserts the parts of every used credit and returns
|
||||
// the set of credits that were actually stored.
|
||||
//
|
||||
// A credit is stored whole or not at all. If any of its artists has no
|
||||
// MBID -- which should not happen, the dump being self-consistent, but
|
||||
// would leave a part that cannot be navigated to -- the credit is
|
||||
// dropped and the entity falls back to explore_index's single artist,
|
||||
// which is a worse answer rather than a broken one.
|
||||
func (imp *dumpImporter) writeCreditParts(
|
||||
ctx context.Context, tx *sql.Tx, scan *creditScan,
|
||||
) (map[int32]struct{}, error) {
|
||||
stmt, err := tx.PrepareContext(ctx,
|
||||
`INSERT INTO artist_credit_part
|
||||
(credit_id, position, artist_mbid, credited_name, join_phrase)
|
||||
VALUES (?, ?, ?, ?, ?)`,
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("credit import: prepare part insert: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = stmt.Close() }()
|
||||
|
||||
written := make(map[int32]struct{}, len(scan.used))
|
||||
|
||||
for credit := range scan.used {
|
||||
parts := scan.parts[credit]
|
||||
if len(parts) < 2 {
|
||||
// artist_credit said more than one artist and
|
||||
// artist_credit_name did not deliver them. Nothing to
|
||||
// decompose, so leave the entity to its single artist.
|
||||
continue
|
||||
}
|
||||
|
||||
// Position order is the credit's meaning, and the dump is not
|
||||
// obliged to emit it sorted.
|
||||
sort.Slice(parts, func(i, j int) bool {
|
||||
return parts[i].position < parts[j].position
|
||||
})
|
||||
|
||||
resolved := make([][]any, 0, len(parts))
|
||||
ok := true
|
||||
|
||||
for _, part := range parts {
|
||||
gid, found := scan.artistGIDs[part.artistID]
|
||||
if !found {
|
||||
scan.skippedUnknownArtist++
|
||||
ok = false
|
||||
|
||||
break
|
||||
}
|
||||
|
||||
resolved = append(resolved, []any{
|
||||
credit, part.position, gid[:], part.name, part.join,
|
||||
})
|
||||
}
|
||||
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
|
||||
for _, args := range resolved {
|
||||
if _, err := stmt.ExecContext(ctx, args...); err != nil {
|
||||
return nil, fmt.Errorf("credit import: insert part: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
written[credit] = struct{}{}
|
||||
}
|
||||
|
||||
return written, nil
|
||||
}
|
||||
|
||||
// writeCreditRefs points each kept entity at its credit, skipping any
|
||||
// whose credit was not stored so a ref never dangles.
|
||||
func (imp *dumpImporter) writeCreditRefs(
|
||||
ctx context.Context, tx *sql.Tx, scan *creditScan, written map[int32]struct{},
|
||||
) (int, error) {
|
||||
stmt, err := tx.PrepareContext(ctx,
|
||||
"INSERT OR REPLACE INTO artist_credit_ref (mbid, credit_id) VALUES (?, ?)",
|
||||
)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("credit import: prepare ref insert: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = stmt.Close() }()
|
||||
|
||||
count := 0
|
||||
|
||||
for mbid, credit := range scan.refs {
|
||||
if _, stored := written[credit]; !stored {
|
||||
continue
|
||||
}
|
||||
|
||||
id := mbid
|
||||
|
||||
if _, err := stmt.ExecContext(ctx, id[:], credit); err != nil {
|
||||
return 0, fmt.Errorf("credit import: insert ref: %w", err)
|
||||
}
|
||||
|
||||
count++
|
||||
}
|
||||
|
||||
return count, nil
|
||||
}
|
||||
@@ -104,6 +104,7 @@ type dumpImporter struct {
|
||||
|
||||
canonicalBaseURL string
|
||||
listensBaseURL string
|
||||
mbdumpBaseURL string
|
||||
|
||||
// Disk safety floors (fields so tests can relax them).
|
||||
minStartFreeBytes uint64
|
||||
@@ -144,6 +145,7 @@ func newDumpImporter(si *SearchIndex, lb *ListenBrainzClient) (*dumpImporter, er
|
||||
stagingDir: stagingDir,
|
||||
canonicalBaseURL: defaultCanonicalBaseURL,
|
||||
listensBaseURL: defaultListensBaseURL,
|
||||
mbdumpBaseURL: defaultMBDumpBaseURL,
|
||||
minStartFreeBytes: dumpMinStartFreeBytes,
|
||||
abortFreeBytes: dumpAbortFreeBytes,
|
||||
}, nil
|
||||
@@ -171,6 +173,7 @@ func (imp *dumpImporter) run(ctx context.Context) error {
|
||||
// Fast path: rows already assembled, only patch passes remain.
|
||||
if state.Stage == dumpStageAssembled {
|
||||
imp.si.MarkReadyIfPopulated()
|
||||
imp.ensureArtistCredits(ctx)
|
||||
imp.runPatchPasses(ctx)
|
||||
|
||||
if err := ctx.Err(); err != nil {
|
||||
@@ -305,6 +308,11 @@ func (imp *dumpImporter) run(ctx context.Context) error {
|
||||
imp.si.MarkReadyIfPopulated()
|
||||
imp.si.refreshStatusCounts()
|
||||
|
||||
// Multi-artist credits, from a different dump. After the catalog,
|
||||
// because it asks explore_index which entities are worth carrying
|
||||
// credits for.
|
||||
imp.ensureArtistCredits(ctx)
|
||||
|
||||
// Stage 4: API patch passes (idempotent).
|
||||
imp.runPatchPasses(ctx)
|
||||
|
||||
|
||||
+14
-1
@@ -177,12 +177,25 @@ func run(o opts) error {
|
||||
|
||||
complete := svc.IndexImportComplete() && !errors.Is(err, errIncomplete)
|
||||
|
||||
// Credits are maintenance, not part of any one mode. They come from
|
||||
// a different dump, they are keyed on entities the catalog already
|
||||
// holds, and a catalog built before the pass existed would otherwise
|
||||
// only gain them from a rebuild — which re-downloads ~205 GB to
|
||||
// re-derive rows it already has. Skipped when the import is not
|
||||
// complete, because there is nothing to key them against yet.
|
||||
creditsAdded := false
|
||||
if complete {
|
||||
creditsAdded = svc.EnsureArtistCredits(context.Background())
|
||||
}
|
||||
|
||||
// "Changed" means there is something new worth publishing, so it is
|
||||
// only ever true for a finished import: a build stamps the listens
|
||||
// series early, long before its rows are assembled, and reporting a
|
||||
// change off that would be a lie about a half-built index.
|
||||
changed := complete &&
|
||||
(svc.IndexBaselineSeries() != seriesBefore || chosen != modeRefresh)
|
||||
(svc.IndexBaselineSeries() != seriesBefore ||
|
||||
chosen != modeRefresh ||
|
||||
creditsAdded)
|
||||
|
||||
report(logger, svc, chosen, complete, changed)
|
||||
|
||||
|
||||
@@ -12,6 +12,7 @@ import (
|
||||
_ "modernc.org/sqlite"
|
||||
|
||||
"yellowjacket/backend/database"
|
||||
"yellowjacket/backend/datamap"
|
||||
"yellowjacket/backend/system"
|
||||
)
|
||||
|
||||
@@ -53,10 +54,19 @@ func TestRetireLibraryTables(t *testing.T) {
|
||||
CREATE TABLE recordings (id INTEGER PRIMARY KEY, title TEXT);
|
||||
`)
|
||||
|
||||
// The symptom, before the repair: the schema cannot be applied over
|
||||
// a table whose shape has moved on.
|
||||
if _, err := database.NewDB(logger); err == nil {
|
||||
t.Fatal("expected the stale shape to fail to open; it did not")
|
||||
// This used to assert the symptom -- that the schema cannot be
|
||||
// applied over a table whose shape has moved on -- because at the
|
||||
// time nothing repaired it and only this job did. The app-side
|
||||
// repair (backend/database/staleshape.go) now retires a stale
|
||||
// non-authored table before applySchema meets it, so opening
|
||||
// succeeds and the symptom no longer reproduces from here.
|
||||
//
|
||||
// That does not make retireLibraryTables redundant, and the rest of
|
||||
// this test is why: the app-side repair only removes what is *stale*,
|
||||
// while this database wants its library half gone entirely, healthy
|
||||
// or not, because nothing here scans, plays or authors.
|
||||
if _, err := database.NewDB(logger); err != nil {
|
||||
t.Fatalf("the app-side repair should have opened this: %v", err)
|
||||
}
|
||||
|
||||
if err := retireLibraryTables(context.Background(), logger); err != nil {
|
||||
@@ -116,3 +126,186 @@ func count(t *testing.T, dbPath, query string) int {
|
||||
|
||||
return n
|
||||
}
|
||||
|
||||
// TestTheCatalogSurvivesAStaleShape is the accident written down.
|
||||
//
|
||||
// The app repairs a stale Cache table by dropping it: its catalog is
|
||||
// downloaded, so a wrong shape costs about a minute of re-fetching and
|
||||
// keeping it costs every Explore read. Applied here that rule is
|
||||
// catastrophic — this database is what the artifact is *cut from*, so
|
||||
// there is nothing to re-fetch and the only way back is the ~205 GB
|
||||
// dump stream the /cache volume exists to avoid.
|
||||
//
|
||||
// It shipped without that distinction and dropped the real CI catalog
|
||||
// on the first run:
|
||||
//
|
||||
// retiring a table ... table=explore_index
|
||||
// reason="column entity_type is TEXT, schema declares INTEGER"
|
||||
// index maintenance mode=build reason="no completed import yet"
|
||||
//
|
||||
// The mismatch was real: that database is deliberately kept in the
|
||||
// older encoding, which `fix(indexexport): read an index older than the
|
||||
// binary` exists to tolerate. So the shape will not match, every run,
|
||||
// by design — and the catalog must survive it anyway.
|
||||
func TestTheCatalogSurvivesAStaleShape(t *testing.T) {
|
||||
logger := slog.New(slog.DiscardHandler)
|
||||
|
||||
t.Setenv("YJ_HOME", t.TempDir())
|
||||
|
||||
dataDir, err := system.GetUserDataDirPath()
|
||||
if err != nil {
|
||||
t.Fatalf("resolve data dir: %v", err)
|
||||
}
|
||||
|
||||
dbPath := filepath.Join(dataDir, "yj.db")
|
||||
|
||||
if _, err := database.NewDB(logger); err != nil {
|
||||
t.Fatalf("first open: %v", err)
|
||||
}
|
||||
|
||||
// The shape the real index database is in: every current column,
|
||||
// but the ids and the entity type still text. That is what the
|
||||
// exporter's backward-compatibility fix tolerates, and it is what
|
||||
// the repair saw and called stale.
|
||||
exec(t, dbPath, `
|
||||
DROP TABLE explore_index;
|
||||
CREATE TABLE explore_index (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
entity_type TEXT NOT NULL,
|
||||
mbid TEXT NOT NULL,
|
||||
title TEXT NOT NULL,
|
||||
artist_name TEXT NOT NULL,
|
||||
artist_mbid TEXT NOT NULL,
|
||||
aliases TEXT NOT NULL DEFAULT '',
|
||||
popularity INTEGER NOT NULL DEFAULT 0,
|
||||
listener_count INTEGER NOT NULL DEFAULT 0,
|
||||
duration INTEGER NOT NULL DEFAULT 0,
|
||||
caa_release_mbid TEXT NOT NULL DEFAULT '',
|
||||
release_name TEXT NOT NULL DEFAULT '',
|
||||
primary_type TEXT NOT NULL DEFAULT '',
|
||||
secondary_types TEXT NOT NULL DEFAULT '',
|
||||
release_date TEXT NOT NULL DEFAULT '',
|
||||
total_tracks INTEGER NOT NULL DEFAULT 0,
|
||||
artist_type TEXT NOT NULL DEFAULT '',
|
||||
country TEXT NOT NULL DEFAULT '',
|
||||
disambiguation TEXT NOT NULL DEFAULT '',
|
||||
sort_name TEXT NOT NULL DEFAULT '',
|
||||
in_library INTEGER NOT NULL DEFAULT 0,
|
||||
is_similar INTEGER NOT NULL DEFAULT 0,
|
||||
local_artist_id INTEGER,
|
||||
local_release_group_id INTEGER,
|
||||
local_recording_id INTEGER,
|
||||
discog_fetched INTEGER NOT NULL DEFAULT 0,
|
||||
UNIQUE(mbid)
|
||||
);
|
||||
INSERT INTO explore_index
|
||||
(entity_type, mbid, title, artist_name, artist_mbid)
|
||||
VALUES ('artist', 'a-b-c', 'A Catalog Row', 'An Artist', 'd-e-f');
|
||||
`)
|
||||
|
||||
if _, err := database.NewDB(logger); err != nil {
|
||||
t.Fatalf("open with a stale catalog shape: %v", err)
|
||||
}
|
||||
|
||||
if got := count(t, dbPath, "SELECT COUNT(*) FROM explore_index"); got != 1 {
|
||||
t.Fatalf(
|
||||
"explore_index rows = %d, want 1 — the catalog was retired, "+
|
||||
"which costs this database a ~205GB rebuild",
|
||||
got,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// TestNoCacheTableIsRetiredHere is the general form of the accident
|
||||
// above, and it exists because the specific one is not the risk.
|
||||
//
|
||||
// `TestTheCatalogSurvivesAStaleShape` pins one table in one wrong shape,
|
||||
// which is the failure that happened. What cost the ~205 GB was not that
|
||||
// shape: it was a destructive repair added to `database.NewDB` -- the
|
||||
// one chokepoint every binary in this project shares -- without asking
|
||||
// which binary it was running in. The next such repair will have a
|
||||
// different name and a different reason, and this database still cannot
|
||||
// afford it.
|
||||
//
|
||||
// So the assertion is about the *outcome* rather than the mechanism: put
|
||||
// every Cache table in a shape the schema has certainly moved past, open
|
||||
// the database the way cmd/indexbuild does, and require that all of them
|
||||
// are still there afterwards. Any future repair that drops one fails
|
||||
// here regardless of how it decides to.
|
||||
//
|
||||
// Two things about it are deliberate.
|
||||
//
|
||||
// The table list comes from `datamap.ByKind(Cache)` rather than being
|
||||
// written out, so a Cache table added next year is covered by this test
|
||||
// on the day it is added -- the same reason `TestCatalogCoversSchema`
|
||||
// reads the schema instead of a list.
|
||||
//
|
||||
// And `NewDB` returning an error is *accepted*, because that is the
|
||||
// trade the fix documents: with Cache tables no longer rebuilt here, a
|
||||
// shape the schema moved past now fails this job loudly instead of
|
||||
// silently costing it a day of downloading. Loud is fine. Gone is not.
|
||||
func TestNoCacheTableIsRetiredHere(t *testing.T) {
|
||||
logger := slog.New(slog.DiscardHandler)
|
||||
|
||||
t.Setenv("YJ_HOME", t.TempDir())
|
||||
|
||||
dataDir, err := system.GetUserDataDirPath()
|
||||
if err != nil {
|
||||
t.Fatalf("resolve data dir: %v", err)
|
||||
}
|
||||
|
||||
dbPath := filepath.Join(dataDir, "yj.db")
|
||||
|
||||
if _, err := database.NewDB(logger); err != nil {
|
||||
t.Fatalf("first open: %v", err)
|
||||
}
|
||||
|
||||
// An FTS table is four shadow tables and cannot be given a "wrong
|
||||
// shape" meaningfully; the repair skips them for the same reason and
|
||||
// retires them with their parent, which the parents below cover.
|
||||
var cache []string
|
||||
|
||||
for _, table := range datamap.ByKind(datamap.Cache) {
|
||||
if table.FTS {
|
||||
continue
|
||||
}
|
||||
|
||||
cache = append(cache, table.Name)
|
||||
}
|
||||
|
||||
if len(cache) == 0 {
|
||||
t.Fatal("no Cache tables to check: the datamap or this test is wrong")
|
||||
}
|
||||
|
||||
for _, name := range cache {
|
||||
// A shape nothing in the current schema describes. What matters
|
||||
// is only that it disagrees; the real mismatch was one column's
|
||||
// type.
|
||||
exec(t, dbPath, `
|
||||
DROP TABLE IF EXISTS `+name+`;
|
||||
CREATE TABLE `+name+` (id INTEGER PRIMARY KEY, moved_past TEXT);
|
||||
INSERT INTO `+name+` (moved_past) VALUES ('irreplaceable');
|
||||
`)
|
||||
}
|
||||
|
||||
// The error is not the assertion: see the note above.
|
||||
_, _ = database.NewDB(logger)
|
||||
|
||||
for _, name := range cache {
|
||||
rows := count(t, dbPath,
|
||||
`SELECT count(*) FROM sqlite_master WHERE type = 'table' AND name = '`+name+`'`)
|
||||
if rows == 0 {
|
||||
t.Errorf("%s was retired: in this database a Cache table is derived, "+
|
||||
"not downloaded, and dropping one costs the ~205 GB dump stream", name)
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
// Present but emptied is the same loss wearing a different
|
||||
// shape: SQLite does an implicit DELETE before a DROP, and a
|
||||
// repair that recreated the table would look identical here.
|
||||
if n := count(t, dbPath, `SELECT count(*) FROM `+name); n == 0 {
|
||||
t.Errorf("%s survived but was emptied", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -207,6 +207,26 @@ func createSchema(db *sql.DB) error {
|
||||
key TEXT PRIMARY KEY,
|
||||
value TEXT NOT NULL
|
||||
)`,
|
||||
// Multi-artist credits. Shipped as their own tables rather than
|
||||
// as an explore_index column because a credit is a variable
|
||||
// number of ordered parts, and because credits are *shared* --
|
||||
// an album's tracks by one artist reference one credit, which is
|
||||
// what keeps this to a few hundred thousand rows.
|
||||
//
|
||||
// An importer that predates these reads an artifact without
|
||||
// them; artifactHasCredits is what asks.
|
||||
`CREATE TABLE core.artist_credit_part (
|
||||
credit_id INTEGER NOT NULL,
|
||||
position INTEGER NOT NULL,
|
||||
artist_mbid BLOB NOT NULL,
|
||||
credited_name TEXT NOT NULL,
|
||||
join_phrase TEXT NOT NULL DEFAULT '',
|
||||
PRIMARY KEY (credit_id, position)
|
||||
) WITHOUT ROWID`,
|
||||
`CREATE TABLE core.artist_credit_ref (
|
||||
mbid BLOB NOT NULL PRIMARY KEY,
|
||||
credit_id INTEGER NOT NULL
|
||||
) WITHOUT ROWID`,
|
||||
}
|
||||
|
||||
for _, stmt := range stmts {
|
||||
@@ -283,6 +303,63 @@ func copyRows(db *sql.DB, artists, perArtistRGs, perArtistRecs int) error {
|
||||
fmt.Printf(" %-15s %d\n", sel.label+":", n)
|
||||
}
|
||||
|
||||
return copyCredits(db)
|
||||
}
|
||||
|
||||
// copyCredits ships the credit decomposition for the entities that made
|
||||
// it into the artifact, and only those.
|
||||
//
|
||||
// The refs go first and the parts follow *from* the refs, so a credit is
|
||||
// carried only if something in the artifact points at it. The source
|
||||
// index holds credits for every catalog entity, while the artifact is a
|
||||
// windowed subset -- copying all of them would carry a large table most
|
||||
// of which nothing in the artifact can reach.
|
||||
//
|
||||
// A source index built before the credit pass simply has no rows here,
|
||||
// which is not an error: the artifact then carries the tables empty, and
|
||||
// every credit falls back to its single artist exactly as before.
|
||||
func copyCredits(db *sql.DB) error {
|
||||
// Asked, not assumed. A source index built before the credit pass
|
||||
// has no such table, and "no such table" would fail an export whose
|
||||
// catalog is otherwise complete.
|
||||
for _, table := range []string{"artist_credit_ref", "artist_credit_part"} {
|
||||
var n int
|
||||
|
||||
if err := db.QueryRow(
|
||||
`SELECT COUNT(*) FROM main.sqlite_master
|
||||
WHERE type = 'table' AND name = ?`, table,
|
||||
).Scan(&n); err != nil {
|
||||
return fmt.Errorf("probe %s: %w", table, err)
|
||||
}
|
||||
|
||||
if n == 0 {
|
||||
fmt.Printf(" %-15s none in source\n", "credits:")
|
||||
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
refs, err := insertSelect(db, `
|
||||
INSERT INTO core.artist_credit_ref (mbid, credit_id)
|
||||
SELECT r.mbid, r.credit_id
|
||||
FROM main.artist_credit_ref r
|
||||
WHERE r.mbid IN (SELECT mbid FROM core.explore_index)`)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
parts, err := insertSelect(db, `
|
||||
INSERT INTO core.artist_credit_part
|
||||
(credit_id, position, artist_mbid, credited_name, join_phrase)
|
||||
SELECT p.credit_id, p.position, p.artist_mbid, p.credited_name, p.join_phrase
|
||||
FROM main.artist_credit_part p
|
||||
WHERE p.credit_id IN (SELECT credit_id FROM core.artist_credit_ref)`)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
fmt.Printf(" %-15s %d refs, %d parts\n", "credits:", refs, parts)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
# The index cache, and why it has a snapshot
|
||||
|
||||
`/srv/yellowjacket/index-cache` on the Gitea host is the `YJ_HOME` the
|
||||
search-index job keeps between runs — `.gitea/workflows/index-artifact.yml`
|
||||
mounts it at `/cache`. It holds the catalog every user eventually
|
||||
downloads, and it is the one database in this project that is
|
||||
**derived rather than downloaded**.
|
||||
|
||||
That is the whole reason this document exists. An install with a broken
|
||||
catalog re-fetches the ~0.6 GB artifact and is fine in a minute. This
|
||||
database *is* what that artifact is cut from, so its only route back is
|
||||
re-streaming the MetaBrainz dumps: hours, at a rate that belongs to
|
||||
someone else's server, holding a runner of capacity 1 the entire time.
|
||||
|
||||
## What happened on 2026-08-17
|
||||
|
||||
A schema repair (`fix(database): retire a table whose shape the schema
|
||||
moved past`) dropped every table whose live shape disagreed with the
|
||||
schema, before `applySchema`. Correct for the app. Applied here it
|
||||
deleted the catalog 19 seconds into the first run:
|
||||
|
||||
```
|
||||
retiring a table ... table=explore_index
|
||||
reason="column entity_type is TEXT, schema declares INTEGER"
|
||||
index maintenance mode=build reason="no completed import yet"
|
||||
```
|
||||
|
||||
The mismatch was real and deliberate: this database is kept in the older
|
||||
text encoding, which `artifactStoresText` and `sourceColumns` exist to
|
||||
tolerate. So it would have been judged stale on *every* run.
|
||||
|
||||
Two things came out of it. `retireStaleCache` is now a build tag —
|
||||
false under `indexbuild`, true in the app — and
|
||||
`TestNoCacheTableIsRetiredHere` asserts the outcome rather than the
|
||||
mechanism, so the next destructive repair fails a test instead of a
|
||||
production volume. And the volume got the snapshot it should always have
|
||||
had, below.
|
||||
|
||||
## Taking snapshots
|
||||
|
||||
```sh
|
||||
scripts/index-cache-snapshot.sh [SOURCE_HOME] [DEST_DIR] [KEEP]
|
||||
```
|
||||
|
||||
Defaults: `/srv/yellowjacket/index-cache`, `/srv/yellowjacket/index-snapshots`,
|
||||
keep 2. On the Gitea host, daily and away from the Monday 04:00 build:
|
||||
|
||||
```
|
||||
30 5 * * * /path/to/index-cache-snapshot.sh >> /var/log/yj-index-snapshot.log 2>&1
|
||||
```
|
||||
|
||||
Three properties worth knowing before trusting it:
|
||||
|
||||
- **It uses `VACUUM INTO`, not `cp`.** The database may be open, and a
|
||||
byte copy of a live SQLite file is a corrupt file of plausible size.
|
||||
`VACUUM INTO` takes a read lock and writes a consistent, compacted
|
||||
copy; it is safe to run while a build is in progress.
|
||||
- **It does not copy `data/explore-staging`.** That is a resumable
|
||||
checkpoint of work in flight — large, constantly changing, and a build
|
||||
resumes without it. What cannot be cheaply re-derived is the finished
|
||||
catalog, which is in the database.
|
||||
- **It verifies before it rotates.** Each snapshot is reopened and asked
|
||||
for its catalog row count; a run that produces an unreadable or empty
|
||||
file fails loudly, deletes its own output, and leaves the previous
|
||||
snapshots alone. Both paths are exercised, not assumed.
|
||||
|
||||
## Restoring
|
||||
|
||||
Stop anything that might be using the volume first — the job holds it
|
||||
for the length of a build, and the concurrency group (`search-index`)
|
||||
means a queued run will start the moment one ends.
|
||||
|
||||
```sh
|
||||
cd /srv/yellowjacket
|
||||
mv index-cache/data/yj.db index-cache/data/yj.db.broken # keep it until you are sure
|
||||
cp index-snapshots/yj-index-<stamp>.db index-cache/data/yj.db
|
||||
chown --reference=index-cache/data/yj.db.broken index-cache/data/yj.db
|
||||
```
|
||||
|
||||
Then dispatch the workflow with `mode=auto`. A restored snapshot is
|
||||
older than the dumps, so `indexbuild` resolves to `refresh` and folds in
|
||||
the incremental listens since — which is minutes, not hours.
|
||||
|
||||
Two notes on what a restore does *not* need. The staging directory can
|
||||
be deleted; it will be rebuilt if a build is needed. And the published
|
||||
artifact is untouched by any of this: users keep downloading the last
|
||||
good one until a run reports `complete=true` and `changed=true`
|
||||
republishes.
|
||||
|
||||
## The trade this leaves open
|
||||
|
||||
With Cache tables no longer retired under `indexbuild`, a future
|
||||
`explore_index` column change will fail this job **loudly** — at
|
||||
`applySchema`, or at the first query naming the column — rather than
|
||||
silently rebuilding. That is the right default: loud is recoverable and
|
||||
a silent day of downloading is not. It does mean the next schema change
|
||||
touching `explore_index` needs a deliberate plan for this one database:
|
||||
take a snapshot, apply the change to a copy, or accept a rebuild
|
||||
knowingly.
|
||||
@@ -12,6 +12,7 @@ export {
|
||||
|
||||
export type {
|
||||
AlbumCompleteFunc,
|
||||
CreditPart,
|
||||
IndexStatus,
|
||||
LBSimilarArtist,
|
||||
LBTopRecording,
|
||||
|
||||
@@ -7,6 +7,23 @@
|
||||
*/
|
||||
export type AlbumCompleteFunc = any;
|
||||
|
||||
/**
|
||||
* CreditPart is one credited artist within a credit, in credit order.
|
||||
*
|
||||
* CreditedName is the name *as credited*, which is not the artist's own
|
||||
* name: MusicBrainz credits "Snoop Dogg" on a track by the artist
|
||||
* called "Snoop Doggy Dogg". Display uses it; navigation uses
|
||||
* ArtistMBID. JoinPhrase is the literal connector that follows this
|
||||
* part, so a credit renders by concatenation and never by searching a
|
||||
* name inside a credit string.
|
||||
*/
|
||||
export interface CreditPart {
|
||||
"position": number;
|
||||
"artistMbid": string;
|
||||
"creditedName": string;
|
||||
"joinPhrase": string;
|
||||
}
|
||||
|
||||
/**
|
||||
* IndexStatus is the full index build status, exposed to the frontend.
|
||||
*/
|
||||
|
||||
@@ -225,6 +225,18 @@ export function GetCandidateThumbnail(releaseMBID: string, releaseGroupMBID: str
|
||||
return $Call.ByID(1946932424, releaseMBID, releaseGroupMBID);
|
||||
}
|
||||
|
||||
/**
|
||||
* GetCredits is the bound form: the frontend asks for a tracklist's
|
||||
* worth of MBIDs at once rather than one per row.
|
||||
*
|
||||
* Batched for the reason every other per-row backend question here is:
|
||||
* asking on hover or on render turns a list into N IPC round trips, and
|
||||
* this one is asked about every row of every list in the app.
|
||||
*/
|
||||
export function GetCredits(mbids: string[] | null): $CancellablePromise<{ [_ in string]?: $models.CreditPart[] | null } | null> {
|
||||
return $Call.ByID(225964099, mbids);
|
||||
}
|
||||
|
||||
/**
|
||||
* GetExploreShelves builds the page Explore shows before a query.
|
||||
*
|
||||
|
||||
@@ -52,7 +52,8 @@ import {
|
||||
} from '@utils/context-menu-controller.js';
|
||||
import type { ContextMenuHost } from '@utils/context-menu-controller.js';
|
||||
import { FavoritesController } from '@store/controllers/favorites-controller';
|
||||
import { artistLink, exploreLinkStyles } from '../../utils/explore-link';
|
||||
import { creditLink, exploreLinkStyles } from '../../utils/explore-link';
|
||||
import { creditStore } from '@store/credit-store';
|
||||
import {
|
||||
createAlbumArtDragImage,
|
||||
createDragImage,
|
||||
@@ -425,8 +426,18 @@ export class CoverGrid
|
||||
* Lifecycle
|
||||
* ==================================================================== */
|
||||
|
||||
/** Unsubscribes the credit-arrival repaint. */
|
||||
private creditsUnsub?: () => void;
|
||||
|
||||
override connectedCallback() {
|
||||
super.connectedCallback();
|
||||
|
||||
this.creditsUnsub = creditStore.subscribe(() => {
|
||||
this.requestUpdate();
|
||||
// Two virtualizers when the grid is split; both draw rows.
|
||||
this.renderRoot?.querySelectorAll('lit-virtualizer')
|
||||
.forEach((v) => (v as unknown as { requestUpdate(): void }).requestUpdate());
|
||||
});
|
||||
this.restoreSortPreferences();
|
||||
this.loadAlbums();
|
||||
|
||||
@@ -441,6 +452,8 @@ export class CoverGrid
|
||||
|
||||
override disconnectedCallback() {
|
||||
super.disconnectedCallback();
|
||||
this.creditsUnsub?.();
|
||||
this.creditsUnsub = undefined;
|
||||
|
||||
this.removeEventListener(
|
||||
'error',
|
||||
@@ -1820,7 +1833,7 @@ export class CoverGrid
|
||||
class="artist-name"
|
||||
title="${album.ArtistName}"
|
||||
>
|
||||
${artistLink(album.ArtistName, album.ArtistMBID ?? '')}
|
||||
${creditLink(creditStore.credits(album.MBID), album.ArtistName, album.ArtistMBID ?? '')}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -20,7 +20,8 @@ type MBRelease = explore.MBRelease;
|
||||
type MBTrack = explore.MBTrack;
|
||||
import { exploreCache } from '../../store/explore-cache';
|
||||
import { libraryStore } from '../../store/library-store';
|
||||
import { artistLink, exploreLinkStyles } from '../../utils/explore-link';
|
||||
import { creditLink, exploreLinkStyles } from '../../utils/explore-link';
|
||||
import { creditStore } from '@store/credit-store';
|
||||
import { describeError } from '../../utils/describe-error';
|
||||
import { EventsOn } from '@runtime/runtime';
|
||||
import { Events } from '../../events';
|
||||
@@ -704,8 +705,15 @@ export class ExploreAlbumDetails extends LitElement implements ContextMenuHost {
|
||||
* BrowseReleases fetch never signals readiness. */
|
||||
private releasesFallbackTimer?: number;
|
||||
|
||||
/** Unsubscribes the credit-arrival repaint. */
|
||||
private creditsUnsub?: () => void;
|
||||
|
||||
override connectedCallback() {
|
||||
super.connectedCallback();
|
||||
|
||||
this.creditsUnsub = creditStore.subscribe(() => {
|
||||
this.requestUpdate();
|
||||
});
|
||||
if (this.releaseGroupMBID || this.localAlbumId) {
|
||||
void this.loadAllData();
|
||||
}
|
||||
@@ -760,6 +768,8 @@ export class ExploreAlbumDetails extends LitElement implements ContextMenuHost {
|
||||
|
||||
override disconnectedCallback() {
|
||||
super.disconnectedCallback();
|
||||
this.creditsUnsub?.();
|
||||
this.creditsUnsub = undefined;
|
||||
this.downloadUnsub?.();
|
||||
this.downloadUnsub = null;
|
||||
this.unsubReleasesReady?.();
|
||||
@@ -2850,7 +2860,11 @@ export class ExploreAlbumDetails extends LitElement implements ContextMenuHost {
|
||||
return html`
|
||||
${artist
|
||||
? html`<div class="album-artist">
|
||||
${artistLink(artist, artistMbid)}
|
||||
${creditLink(
|
||||
creditStore.credits(this.releaseGroupMBID),
|
||||
artist,
|
||||
artistMbid,
|
||||
)}
|
||||
</div>`
|
||||
: nothing}
|
||||
${metaParts.length > 0
|
||||
|
||||
@@ -16,7 +16,8 @@ import { exploreCache, ARTIST_IMAGE_CACHE_LIMIT } from '../../store/explore-cach
|
||||
import { queueStore } from '../../store/queue-store';
|
||||
import { notificationStore } from '../../store/notification-store';
|
||||
import '../notifications/inline-notice';
|
||||
import { artistLink, trackLink, exploreLinkStyles } from '../../utils/explore-link';
|
||||
import { creditLink, trackLink, exploreLinkStyles } from '../../utils/explore-link';
|
||||
import { creditStore } from '@store/credit-store';
|
||||
import { describeError } from '../../utils/describe-error';
|
||||
import '@awesome.me/webawesome/dist/components/icon/icon.js';
|
||||
import '../library-status-indicator/library-status-indicator.js';
|
||||
@@ -774,6 +775,14 @@ export class ExploreView extends ViewLifecycleMixin(LitElement) implements Conte
|
||||
}
|
||||
|
||||
protected override onViewActivate(): void {
|
||||
// A cached primary view, so this is torn down on the way out
|
||||
// rather than on disconnect — which never fires here.
|
||||
this.whileActive(
|
||||
creditStore.subscribe(() => {
|
||||
this.requestUpdate();
|
||||
}),
|
||||
);
|
||||
|
||||
// Fetched on arrival rather than on connect: this is a cached
|
||||
// primary view, created and warmed at startup, so a fetch there
|
||||
// is three catalog queries every user pays for whether or not
|
||||
@@ -2193,7 +2202,7 @@ export class ExploreView extends ViewLifecycleMixin(LitElement) implements Conte
|
||||
<div class="album-title" title="${rg.title}">
|
||||
${rg.title}
|
||||
</div>
|
||||
<div class="album-artist">${artistLink(rg.artistCredit, rg.artistMbid ?? '')}</div>
|
||||
<div class="album-artist">${creditLink(creditStore.credits(rg.mbid), rg.artistCredit, rg.artistMbid ?? '')}</div>
|
||||
<div class="album-meta">
|
||||
<div class="album-meta-text">
|
||||
${rg.primaryType
|
||||
@@ -2255,7 +2264,7 @@ export class ExploreView extends ViewLifecycleMixin(LitElement) implements Conte
|
||||
${trackLink(r.title, r.releaseName ?? '', r.releaseGroupMbid ?? '', r.mbid)}
|
||||
</div>
|
||||
<div class="track-artist">
|
||||
${artistLink(r.artistCredit, r.artistMbid ?? '')}
|
||||
${creditLink(creditStore.credits(r.mbid), r.artistCredit, r.artistMbid ?? '')}
|
||||
</div>
|
||||
</div>
|
||||
<div class="track-meta">
|
||||
|
||||
@@ -5,11 +5,12 @@ import '../audio-player/controls/player-controls';
|
||||
import '../audio-player/seekbar/seek-bar';
|
||||
import '../audio-player/volume-control/volume-control';
|
||||
import {
|
||||
artistLink,
|
||||
creditLink,
|
||||
albumLink,
|
||||
exploreLinkStyles,
|
||||
} from '@utils/explore-link';
|
||||
import { PlayerController } from '@store/controllers/player-controller';
|
||||
import { creditStore } from '@store/credit-store';
|
||||
import { FavoritesController } from '@store/controllers/favorites-controller';
|
||||
import { designTokens } from '../../styles/tokens.css';
|
||||
import { srOnly } from '../../styles/sr-only.css';
|
||||
@@ -36,6 +37,22 @@ import { srOnly } from '../../styles/sr-only.css';
|
||||
@customElement('now-playing-view')
|
||||
export class NowPlayingView extends LitElement {
|
||||
private player = new PlayerController(this);
|
||||
|
||||
/** Unsubscribes the credit-arrival repaint. */
|
||||
private creditsUnsub?: () => void;
|
||||
|
||||
override connectedCallback(): void {
|
||||
super.connectedCallback();
|
||||
// Credits arrive after the track does, so the name this view is
|
||||
// already showing has to be re-rendered when they land.
|
||||
this.creditsUnsub = creditStore.subscribe(() => this.requestUpdate());
|
||||
}
|
||||
|
||||
override disconnectedCallback(): void {
|
||||
super.disconnectedCallback();
|
||||
this.creditsUnsub?.();
|
||||
this.creditsUnsub = undefined;
|
||||
}
|
||||
private favCtrl = new FavoritesController(this);
|
||||
|
||||
static override styles = [designTokens, srOnly, exploreLinkStyles, css`
|
||||
@@ -262,7 +279,11 @@ export class NowPlayingView extends LitElement {
|
||||
${track.title || track.fileName}
|
||||
</h2>
|
||||
<p class="artist">
|
||||
${artistLink(track.artist, track.artistMbid)}
|
||||
${creditLink(
|
||||
creditStore.credits(track.recordingMbid),
|
||||
track.artist,
|
||||
track.artistMbid,
|
||||
)}
|
||||
</p>
|
||||
${track.album
|
||||
? html`<p class="album">
|
||||
|
||||
@@ -4,7 +4,7 @@ import '@awesome.me/webawesome/dist/components/icon/icon.js';
|
||||
import '@awesome.me/webawesome/dist/components/popup/popup.js';
|
||||
import type WaPopup from '@awesome.me/webawesome/dist/components/popup/popup.js';
|
||||
import {
|
||||
artistLink,
|
||||
creditLink,
|
||||
trackLink,
|
||||
exploreLinkStyles,
|
||||
} from '@utils/explore-link';
|
||||
@@ -14,6 +14,7 @@ import {
|
||||
navigateToQueueSource,
|
||||
} from '@utils/queue-source-link';
|
||||
import { PlayerController } from '@store/controllers/player-controller';
|
||||
import { creditStore } from '@store/credit-store';
|
||||
import { QueueController } from '@store/controllers/queue-controller';
|
||||
import { FavoritesController } from '@store/controllers/favorites-controller';
|
||||
import { designTokens } from '../../styles/tokens.css';
|
||||
@@ -325,6 +326,9 @@ export class NowPlaying extends LitElement {
|
||||
}
|
||||
`];
|
||||
|
||||
/** Unsubscribes the credit-arrival repaint. */
|
||||
private creditsUnsub?: () => void;
|
||||
|
||||
override connectedCallback() {
|
||||
super.connectedCallback();
|
||||
this.loadScrollMode();
|
||||
@@ -341,10 +345,21 @@ export class NowPlaying extends LitElement {
|
||||
this.geometryDirty = true;
|
||||
this.requestUpdate();
|
||||
});
|
||||
|
||||
// A credit arriving changes the rendered text, and the marquee
|
||||
// measures that text — so this is a geometry change, not just a
|
||||
// repaint. Saying so is what stops the bar scrolling to the
|
||||
// old width.
|
||||
this.creditsUnsub = creditStore.subscribe(() => {
|
||||
this.geometryDirty = true;
|
||||
this.requestUpdate();
|
||||
});
|
||||
}
|
||||
|
||||
override disconnectedCallback() {
|
||||
super.disconnectedCallback();
|
||||
this.creditsUnsub?.();
|
||||
this.creditsUnsub = undefined;
|
||||
// A drag interrupted by the bar going away still has to clean up.
|
||||
this.attachDragListeners(false);
|
||||
window.removeEventListener(SCROLL_CHANGE_EVENT, this.handleScrollModeEvent);
|
||||
@@ -483,7 +498,7 @@ export class NowPlaying extends LitElement {
|
||||
@mouseleave=${this.handleArtistMouseLeave}
|
||||
@transitionend=${() => this.onScrollCycleEnd('artist')}
|
||||
>
|
||||
<span class="scroll-content">${artistLink(track.artist, track.artistMbid) || 'Unknown Artist'}</span>
|
||||
<span class="scroll-content">${creditLink(creditStore.credits(track.recordingMbid), track.artist, track.artistMbid) || 'Unknown Artist'}</span>
|
||||
</span>
|
||||
${describeQueueSource(this.queue.source)
|
||||
? html`
|
||||
|
||||
@@ -24,6 +24,7 @@ import type * as playlist from '@go/playlist/models.js';
|
||||
import { EventsOn } from '@runtime/runtime';
|
||||
import { Events } from '../../events';
|
||||
import { queueStore } from '@store/queue-store';
|
||||
import { creditStore } from '@store/credit-store';
|
||||
import { PlayerController } from '@store/controllers/player-controller';
|
||||
import { SearchController } from '@store/controllers/search-controller';
|
||||
import { SelectionController } from '@utils/selection-controller';
|
||||
@@ -61,7 +62,8 @@ import '@components/duplicate-tracks-dialog/duplicate-tracks-dialog.js';
|
||||
import type { DuplicateTracksDialog } from '@components/duplicate-tracks-dialog/duplicate-tracks-dialog.js';
|
||||
import { formatMilliseconds } from '@utils/time';
|
||||
import {
|
||||
artistLink,
|
||||
creditLink,
|
||||
creditText,
|
||||
albumLink,
|
||||
trackLink,
|
||||
exploreLinkStyles,
|
||||
@@ -106,6 +108,9 @@ export class PlaylistDetails
|
||||
* rather than guessed. Without the hint the flow layout's 100 px
|
||||
* default drives constant scroll-error correction, which reads as
|
||||
* the list jumping under the pointer. */
|
||||
/** Unsubscribes the credit-arrival repaint. */
|
||||
private creditsUnsub?: () => void;
|
||||
|
||||
@query('lit-virtualizer')
|
||||
private virtualizer?: LitVirtualizer;
|
||||
|
||||
@@ -224,6 +229,14 @@ export class PlaylistDetails
|
||||
|
||||
override connectedCallback() {
|
||||
super.connectedCallback();
|
||||
|
||||
// Credits arrive after the rows that asked for them, and a
|
||||
// virtualizer repaints from its *own* properties — a host
|
||||
// update alone leaves the rows exactly as they were.
|
||||
this.creditsUnsub = creditStore.subscribe(() => {
|
||||
this.requestUpdate();
|
||||
this.virtualizer?.requestUpdate();
|
||||
});
|
||||
this.loadTracks();
|
||||
|
||||
this.tracksChangedCleanup = EventsOn(
|
||||
@@ -248,6 +261,8 @@ export class PlaylistDetails
|
||||
|
||||
override disconnectedCallback() {
|
||||
super.disconnectedCallback();
|
||||
this.creditsUnsub?.();
|
||||
this.creditsUnsub = undefined;
|
||||
|
||||
if (this.tracksChangedCleanup) {
|
||||
this.tracksChangedCleanup();
|
||||
@@ -1575,7 +1590,7 @@ export class PlaylistDetails
|
||||
: nothing}
|
||||
</div>
|
||||
<span class="cell col-title" title="${track.Title || track.FilePath}">${trackLink(track.Title, track.Album, track.ReleaseGroupMBID, track.RecordingMBID, undefined, track.Artist) || track.FilePath}</span>
|
||||
<span class="cell col-artist" title="${track.Artist}">${artistLink(track.Artist, track.ArtistMBID)}</span>
|
||||
<span class="cell col-artist" title="${creditText(creditStore.credits(track.RecordingMBID), track.Artist)}">${creditLink(creditStore.credits(track.RecordingMBID), track.Artist, track.ArtistMBID)}</span>
|
||||
<span class="cell col-album" title="${track.Album}">${albumLink(track.Album, track.ReleaseGroupMBID, undefined, track.Artist)}</span>
|
||||
<span class="cell col-duration">${formatMilliseconds(track.Duration)}</span>`}
|
||||
</div>
|
||||
|
||||
@@ -12,6 +12,7 @@ import '@awesome.me/webawesome/dist/components/popup/popup.js';
|
||||
import type WaPopup from '@awesome.me/webawesome/dist/components/popup/popup.js';
|
||||
import '@awesome.me/webawesome/dist/components/dropdown-item/dropdown-item.js';
|
||||
import { QueueController } from '@store/controllers/queue-controller';
|
||||
import { creditStore } from '@store/credit-store';
|
||||
import {
|
||||
describeQueueSource,
|
||||
isQueueSourceNavigable,
|
||||
@@ -55,7 +56,7 @@ import { tracksByFilePath } from '@utils/track-index.js';
|
||||
import type { TrackDetails } from '@components/track-details/track-details.js';
|
||||
import type { CoverArtUrls } from '@components/track-details/track-details.js';
|
||||
import {
|
||||
artistLink,
|
||||
creditLink,
|
||||
trackLink,
|
||||
exploreLinkStyles,
|
||||
} from '@utils/explore-link';
|
||||
@@ -108,6 +109,9 @@ export class QueuePanel
|
||||
@query('#playlist-submenu')
|
||||
private playlistSubmenuPopup!: WaPopup;
|
||||
|
||||
/** Unsubscribes the credit-arrival repaint. */
|
||||
private creditsUnsub?: () => void;
|
||||
|
||||
@query('lit-virtualizer')
|
||||
private virtualizer!: LitVirtualizer;
|
||||
|
||||
@@ -635,6 +639,14 @@ export class QueuePanel
|
||||
|
||||
override connectedCallback() {
|
||||
super.connectedCallback();
|
||||
|
||||
// Credits arrive after the rows that asked for them, and a
|
||||
// virtualizer repaints from its *own* properties — a host
|
||||
// update alone leaves the rows exactly as they were.
|
||||
this.creditsUnsub = creditStore.subscribe(() => {
|
||||
this.requestUpdate();
|
||||
this.virtualizer?.requestUpdate();
|
||||
});
|
||||
this.style.setProperty(
|
||||
'--queue-width',
|
||||
`${this.panelWidth}px`,
|
||||
@@ -671,6 +683,8 @@ export class QueuePanel
|
||||
|
||||
override disconnectedCallback() {
|
||||
super.disconnectedCallback();
|
||||
this.creditsUnsub?.();
|
||||
this.creditsUnsub = undefined;
|
||||
document.removeEventListener(
|
||||
'mousemove',
|
||||
this.handleMouseMove,
|
||||
@@ -1675,7 +1689,7 @@ export class QueuePanel
|
||||
${trackLink(title, track.album, track.releaseGroupMbid, track.recordingMbid, undefined, track.artist)}
|
||||
</span>
|
||||
<span class="track-artist" title=${artist}>
|
||||
${artistLink(track.artist, track.artistMbid) || 'Unknown Artist'}
|
||||
${creditLink(creditStore.credits(track.recordingMbid), track.artist, track.artistMbid) || 'Unknown Artist'}
|
||||
</span>
|
||||
</div>
|
||||
<button
|
||||
|
||||
@@ -15,6 +15,7 @@ import {
|
||||
import { EventsOn } from '@runtime/runtime';
|
||||
import { Events } from '../../events';
|
||||
import { queueStore } from '@store/queue-store';
|
||||
import { creditStore } from '@store/credit-store';
|
||||
import { PlayerController } from '@store/controllers/player-controller';
|
||||
import { SearchController } from '@store/controllers/search-controller';
|
||||
import { SelectionController } from '@utils/selection-controller';
|
||||
@@ -51,7 +52,8 @@ import type { CoverArtUrls } from '@components/track-details/track-details.js';
|
||||
import { libraryStore } from '@store/library-store';
|
||||
import { formatMilliseconds } from '@utils/time';
|
||||
import {
|
||||
artistLink,
|
||||
creditLink,
|
||||
creditText,
|
||||
albumLink,
|
||||
trackLink,
|
||||
exploreLinkStyles,
|
||||
@@ -136,6 +138,9 @@ export class SmartPlaylistDetails
|
||||
* rather than guessed: without the hint the flow layout's 100 px
|
||||
* default drives constant scroll-error correction, which reads as
|
||||
* the list jumping under the pointer. */
|
||||
/** Unsubscribes the credit-arrival repaint. */
|
||||
private creditsUnsub?: () => void;
|
||||
|
||||
@query('lit-virtualizer')
|
||||
private virtualizer?: LitVirtualizer;
|
||||
|
||||
@@ -609,6 +614,14 @@ export class SmartPlaylistDetails
|
||||
override connectedCallback() {
|
||||
super.connectedCallback();
|
||||
|
||||
// Credits arrive after the rows that asked for them, and a
|
||||
// virtualizer repaints from its *own* properties — a host
|
||||
// update alone leaves the rows exactly as they were.
|
||||
this.creditsUnsub = creditStore.subscribe(() => {
|
||||
this.requestUpdate();
|
||||
this.virtualizer?.requestUpdate();
|
||||
});
|
||||
|
||||
if (this.autoEdit) {
|
||||
// Skip evaluation for new playlists — go straight to editor.
|
||||
this.autoEdit = false;
|
||||
@@ -649,6 +662,8 @@ export class SmartPlaylistDetails
|
||||
|
||||
override disconnectedCallback() {
|
||||
super.disconnectedCallback();
|
||||
this.creditsUnsub?.();
|
||||
this.creditsUnsub = undefined;
|
||||
|
||||
if (this.playlistDeletedCleanup) {
|
||||
this.playlistDeletedCleanup();
|
||||
@@ -1423,7 +1438,7 @@ export class SmartPlaylistDetails
|
||||
: nothing}
|
||||
</div>
|
||||
<span class="cell col-title" title="${track.Title || track.FilePath}">${trackLink(track.Title, track.Album, track.ReleaseGroupMBID, track.RecordingMBID, undefined, track.Artist) || track.FilePath}</span>
|
||||
<span class="cell col-artist" title="${track.Artist}">${artistLink(track.Artist, track.ArtistMBID)}</span>
|
||||
<span class="cell col-artist" title="${creditText(creditStore.credits(track.RecordingMBID), track.Artist)}">${creditLink(creditStore.credits(track.RecordingMBID), track.Artist, track.ArtistMBID)}</span>
|
||||
<span class="cell col-album" title="${track.Album}">${albumLink(track.Album, track.ReleaseGroupMBID, undefined, track.Artist)}</span>
|
||||
<span class="cell col-duration">${formatMilliseconds(track.Duration)}</span>`}
|
||||
</div>
|
||||
|
||||
@@ -9,7 +9,8 @@ import {
|
||||
} from '@go/explore/service.js';
|
||||
import '../library-status-indicator/library-status-indicator.js';
|
||||
import type { LibraryStatus } from '../library-status-indicator/library-status-indicator.js';
|
||||
import { artistLink, exploreLinkStyles } from '../../utils/explore-link';
|
||||
import { creditLink, exploreLinkStyles } from '../../utils/explore-link';
|
||||
import { creditStore } from '@store/credit-store';
|
||||
import { libraryStatusFor } from '../../utils/library-status';
|
||||
import { downloadStore } from '../../store/download-store';
|
||||
|
||||
@@ -61,14 +62,23 @@ export class TopResultsRow extends LitElement {
|
||||
* the property and never updates this element. One subscription for
|
||||
* the row, not one per card.
|
||||
*/
|
||||
/** Unsubscribes the credit-arrival repaint. */
|
||||
private creditsUnsub?: () => void;
|
||||
|
||||
override connectedCallback(): void {
|
||||
super.connectedCallback();
|
||||
|
||||
this.creditsUnsub = creditStore.subscribe(() => {
|
||||
this.requestUpdate();
|
||||
});
|
||||
this.unsubRequests = downloadStore.subscribe(() =>
|
||||
this.requestUpdate(),
|
||||
);
|
||||
}
|
||||
|
||||
override disconnectedCallback(): void {
|
||||
this.creditsUnsub?.();
|
||||
this.creditsUnsub = undefined;
|
||||
this.unsubRequests?.();
|
||||
this.unsubRequests = undefined;
|
||||
super.disconnectedCallback();
|
||||
@@ -327,7 +337,7 @@ export class TopResultsRow extends LitElement {
|
||||
${artistPart || metaPart
|
||||
? html`<span class="card-subtitle"
|
||||
>${artistPart
|
||||
? artistLink(artistPart, r.artistMbid ?? '')
|
||||
? creditLink(creditStore.credits(r.mbid), artistPart, r.artistMbid ?? '')
|
||||
: nothing}${artistPart && metaPart
|
||||
? ' · '
|
||||
: ''}${metaPart}</span
|
||||
|
||||
@@ -25,6 +25,7 @@ import type { SortOption } from '@components/page-header/page-header';
|
||||
import { TrackListController } from '@store/controllers/tracklist-controller';
|
||||
import { FavoritesController } from '@store/controllers/favorites-controller';
|
||||
import { queueStore } from '@store/queue-store';
|
||||
import { creditStore } from '@store/credit-store';
|
||||
import type { QueueSource } from '@store/queue-store';
|
||||
import { LibraryController } from '@store/controllers/library-controller';
|
||||
import {
|
||||
@@ -40,6 +41,7 @@ import {
|
||||
} from './search-ranking';
|
||||
import {
|
||||
artistLink,
|
||||
creditLink,
|
||||
albumLink,
|
||||
trackLink,
|
||||
exploreLinkStyles,
|
||||
@@ -1339,6 +1341,17 @@ export class TrackList
|
||||
'shortcut:tracklist-delete',
|
||||
this.handleShortcutDelete,
|
||||
);
|
||||
|
||||
// Credits arrive after the rows that asked for them. The
|
||||
// virtualizer produces its rows from its *own* properties, so a
|
||||
// host re-render alone repaints nothing — the same reason a
|
||||
// selection change pushes requestUpdate() into it.
|
||||
this.whileActive(
|
||||
creditStore.subscribe(() => {
|
||||
this.requestUpdate();
|
||||
this.virtualizer?.requestUpdate();
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -2150,7 +2163,15 @@ export class TrackList
|
||||
if (col.id === 'trackName') {
|
||||
display = trackLink(track.TrackName, track.Album, track.ReleaseGroupMBID, track.RecordingMBID, display as any, track.ArtistName);
|
||||
} else if (col.id === 'artistName') {
|
||||
display = artistLink(track.ArtistName, track.ArtistMBID, display as any);
|
||||
// A search term highlights the *flat* credit string,
|
||||
// and mapping those spans onto decomposed parts is a
|
||||
// different problem from rendering the credit. While
|
||||
// filtering, the single link is the honest answer.
|
||||
creditStore.request(track.RecordingMBID);
|
||||
const parts = term ? undefined : creditStore.get(track.RecordingMBID);
|
||||
display = parts && parts.length > 1
|
||||
? creditLink(parts, track.ArtistName, track.ArtistMBID)
|
||||
: artistLink(track.ArtistName, track.ArtistMBID, display as any);
|
||||
} else if (col.id === 'album') {
|
||||
display = albumLink(track.Album, track.ReleaseGroupMBID, display as any, track.ArtistName);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,226 @@
|
||||
/**
|
||||
* Multi-artist credits, keyed by recording MBID.
|
||||
*
|
||||
* A credit is ordered parts and the credit *string* is derived from
|
||||
* them. This store holds the parts for entities that have more than
|
||||
* one credited artist; everything else renders the single link it
|
||||
* always did.
|
||||
*
|
||||
* Three things about it are load-bearing.
|
||||
*
|
||||
* **Absence is an answer, and it is cached as one.** The backend
|
||||
* returns nothing for a single-artist credit, which is the common case
|
||||
* by a wide margin — measured on a real library, 13% of tracks are
|
||||
* multi-artist. Caching only the hits would re-request the other 87%
|
||||
* on every render, forever, which is the same shape as the bug that
|
||||
* made `explore-album-details` ask the backend on hover. A miss is
|
||||
* stored as an empty array: *asked*, not *answered*.
|
||||
*
|
||||
* **The lookup is batched, and coalesced across callers.** Every row
|
||||
* of every tracklist asks this question, and one IPC round trip per row
|
||||
* is how a 5,000-row list becomes unusable. A virtualized list cannot
|
||||
* hand over "the whole list" either — 50,000 rows is 100 queries for
|
||||
* the ~30 on screen. So `request()` is per-row and cheap: it collects
|
||||
* into a pending set and flushes once on the next frame, which turns a
|
||||
* screenful of rows into exactly one call. `ensure()` remains for a
|
||||
* caller that genuinely has a bounded list in hand.
|
||||
*
|
||||
* **It is bounded.** A cache that grows with use is a leak with a
|
||||
* schedule; a browsing afternoon touches far more credits than a
|
||||
* screenful. The cap is entries rather than bytes because a credit is
|
||||
* a handful of short strings, unlike the art caches next door.
|
||||
*/
|
||||
|
||||
import { GetCredits } from '@go/explore/service.js';
|
||||
import type { CreditPart } from '../utils/explore-link';
|
||||
import { LRUMap } from '../utils/lru-map';
|
||||
import { compact } from '../utils/binding';
|
||||
import { registerCacheProbe } from '../utils/cache-stats';
|
||||
|
||||
/**
|
||||
* Entries retained. A credit is ~4 short strings, so this is well
|
||||
* under a megabyte — sized to comfortably exceed any single list the
|
||||
* app renders, because a cap below the visible count evicts rows that
|
||||
* are still on screen and the re-render fetches them straight back.
|
||||
*/
|
||||
export const CREDIT_CACHE_LIMIT = 20_000;
|
||||
|
||||
/** An empty parts array is the negative marker: asked, no decomposition. */
|
||||
type CachedParts = readonly CreditPart[];
|
||||
|
||||
class CreditStore {
|
||||
private cache = new LRUMap<string, CachedParts>(CREDIT_CACHE_LIMIT);
|
||||
|
||||
/** MBIDs with a request in flight, so a re-render does not refetch. */
|
||||
private inFlight = new Set<string>();
|
||||
|
||||
private listeners = new Set<() => void>();
|
||||
|
||||
/** Collected by request(), flushed as one batch on the next frame. */
|
||||
private pending = new Set<string>();
|
||||
|
||||
private flushHandle: number | null = null;
|
||||
|
||||
constructor() {
|
||||
registerCacheProbe('credits', () => ({
|
||||
entries: this.cache.size,
|
||||
chars: this.retainedChars(),
|
||||
limit: CREDIT_CACHE_LIMIT,
|
||||
}));
|
||||
}
|
||||
|
||||
/**
|
||||
* The strings actually retained, counted rather than estimated —
|
||||
* a bound that is only checkable against a guess is not checkable.
|
||||
*/
|
||||
private retainedChars(): number {
|
||||
let total = 0;
|
||||
|
||||
for (const parts of this.cache.values()) {
|
||||
for (const part of parts) {
|
||||
total +=
|
||||
part.creditedName.length +
|
||||
part.joinPhrase.length +
|
||||
part.artistMbid.length;
|
||||
}
|
||||
}
|
||||
|
||||
return total;
|
||||
}
|
||||
|
||||
/**
|
||||
* Subscribe to "some credits arrived".
|
||||
*
|
||||
* Deliberately not per-MBID: a list fetches its rows in one call and
|
||||
* re-renders once, so a fine-grained signal would buy nothing and
|
||||
* cost a listener per row.
|
||||
*/
|
||||
subscribe(fn: () => void): () => void {
|
||||
this.listeners.add(fn);
|
||||
|
||||
return () => this.listeners.delete(fn);
|
||||
}
|
||||
|
||||
/**
|
||||
* The parts for one entity, or undefined when it has not been asked
|
||||
* about yet.
|
||||
*
|
||||
* An entity with a single-artist credit returns an empty array, and
|
||||
* `creditLink` treats fewer than two parts as the fallback — so a
|
||||
* caller does not have to distinguish "not asked" from "one artist"
|
||||
* to render correctly, only to decide whether to ask.
|
||||
*/
|
||||
get(mbid: string | undefined): readonly CreditPart[] | undefined {
|
||||
if (!mbid) return undefined;
|
||||
|
||||
return this.cache.get(mbid);
|
||||
}
|
||||
|
||||
/**
|
||||
* Ask about one entity, joining whatever batch is forming.
|
||||
*
|
||||
* Safe to call from a render: it is a set insert and a scheduled
|
||||
* flush, and an entity already cached or in flight is dropped. The
|
||||
* loop it looks like it might cause does not happen — after a flush
|
||||
* every requested MBID is cached, so the re-render's requests are
|
||||
* all dropped and nothing notifies again.
|
||||
*/
|
||||
request(mbid: string | undefined): void {
|
||||
if (!mbid) return;
|
||||
if (this.cache.has(mbid)) return;
|
||||
if (this.inFlight.has(mbid)) return;
|
||||
if (this.pending.has(mbid)) return;
|
||||
|
||||
this.pending.add(mbid);
|
||||
|
||||
if (this.flushHandle !== null) return;
|
||||
|
||||
// A frame, not a microtask: the point is to collect every row a
|
||||
// virtualizer renders in this pass, and those happen across the
|
||||
// whole update, not within one microtask checkpoint.
|
||||
this.flushHandle = requestAnimationFrame(() => {
|
||||
this.flushHandle = null;
|
||||
|
||||
const batch = [...this.pending];
|
||||
|
||||
this.pending.clear();
|
||||
|
||||
void this.ensure(batch);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Ask and read in one call, for use inside a template.
|
||||
*
|
||||
* A getter with a side effect, deliberately: the alternative is
|
||||
* every call site writing `request(x)` beside `get(x)` and one of
|
||||
* them eventually forgetting, which renders a permanently
|
||||
* single-artist credit that looks exactly like an entity with one
|
||||
* artist. Making the request the same act as the read is what
|
||||
* stops the two drifting apart.
|
||||
*/
|
||||
credits(mbid: string | undefined): readonly CreditPart[] | undefined {
|
||||
this.request(mbid);
|
||||
|
||||
return this.get(mbid);
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch the credits for a list, skipping anything already known or
|
||||
* already being fetched.
|
||||
*
|
||||
* `has` rather than `get` for the membership test: probing must not
|
||||
* mark an entry recently-used, or scrolling past a row would keep
|
||||
* it alive ahead of one actually being rendered.
|
||||
*/
|
||||
async ensure(mbids: readonly (string | undefined)[]): Promise<void> {
|
||||
const wanted = new Set<string>();
|
||||
|
||||
for (const mbid of mbids) {
|
||||
if (!mbid) continue;
|
||||
if (this.cache.has(mbid)) continue;
|
||||
if (this.inFlight.has(mbid)) continue;
|
||||
|
||||
wanted.add(mbid);
|
||||
}
|
||||
|
||||
if (wanted.size === 0) return;
|
||||
|
||||
const batch = [...wanted];
|
||||
|
||||
for (const mbid of batch) this.inFlight.add(mbid);
|
||||
|
||||
try {
|
||||
const found = compact(await GetCredits(batch));
|
||||
|
||||
for (const mbid of batch) {
|
||||
// Every MBID asked for gets an entry, present or not:
|
||||
// the absent ones are the answer "one artist", and not
|
||||
// recording that is what would re-ask forever.
|
||||
this.cache.set(mbid, found[mbid] ?? []);
|
||||
}
|
||||
|
||||
this.notify();
|
||||
} catch (err) {
|
||||
// A credit is an enrichment: without it every name renders
|
||||
// as the single link it did before, which is a worse answer
|
||||
// rather than a broken one. Nothing user-facing is worth
|
||||
// interrupting for, so this stays in the console.
|
||||
console.error('Failed to load artist credits', err);
|
||||
} finally {
|
||||
for (const mbid of batch) this.inFlight.delete(mbid);
|
||||
}
|
||||
}
|
||||
|
||||
/** Drop everything. The tags on disk changed, so credits may have. */
|
||||
invalidate(): void {
|
||||
this.cache = new LRUMap<string, CachedParts>(CREDIT_CACHE_LIMIT);
|
||||
this.notify();
|
||||
}
|
||||
|
||||
private notify(): void {
|
||||
for (const fn of this.listeners) fn();
|
||||
}
|
||||
}
|
||||
|
||||
export const creditStore = new CreditStore();
|
||||
@@ -294,3 +294,81 @@ async function openAlbum(
|
||||
|
||||
navigate(target, detail);
|
||||
}
|
||||
|
||||
/**
|
||||
* One credited artist within a multi-artist credit.
|
||||
*
|
||||
* Mirrors `artist_credit_part` / `file_artists`: the name **as
|
||||
* credited** (which is not the artist's own name — MusicBrainz credits
|
||||
* "Snoop Dogg" on a track by the artist called "Snoop Doggy Dogg"), the
|
||||
* MBID to navigate to, and the literal connector that follows this
|
||||
* part.
|
||||
*/
|
||||
export interface CreditPart {
|
||||
/** The name as credited. Display uses this. */
|
||||
creditedName: string;
|
||||
/** The artist's MusicBrainz ID. Navigation uses this. */
|
||||
artistMbid: string;
|
||||
/** The connector following this part: " feat. ", " & ", ", ", "". */
|
||||
joinPhrase: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Render a credit as links, one per credited artist, with the join
|
||||
* phrases as plain text between them.
|
||||
*
|
||||
* Join phrases are **assembly instructions, not disassembly
|
||||
* instructions**. This concatenates parts; it never searches for a
|
||||
* name inside a credit string. That distinction is the whole point:
|
||||
* the stored credit text may have come from a file's tags while the
|
||||
* parts come from the catalog, and measured on a real library those
|
||||
* disagree for about one in three multi-artist credits ("Skrillex
|
||||
* feat. Swae Lee" tagged against "Skrillex & Swae Lee" upstream). A
|
||||
* search would miss, or match the wrong span. Building from parts,
|
||||
* the link boundaries are known by construction.
|
||||
*
|
||||
* Falls back to `artistLink(fallbackName, fallbackMbid)` — today's
|
||||
* behaviour exactly — when there are no parts. That is the common
|
||||
* case and not a degraded one: a single-artist credit *is* one link,
|
||||
* and a file with no recording MBID or no catalog row has nothing to
|
||||
* decompose. Do not try to split the fallback string; there is
|
||||
* genuinely no information in it to split on.
|
||||
*
|
||||
* @param parts - The credit's parts in position order, if known.
|
||||
* @param fallbackName - The credit as a single string.
|
||||
* @param fallbackMbid - The primary artist's MBID.
|
||||
*/
|
||||
export function creditLink(
|
||||
parts: readonly CreditPart[] | undefined,
|
||||
fallbackName: string,
|
||||
fallbackMbid: string,
|
||||
): TemplateResult | string {
|
||||
// One part is one link, so it is the fallback rather than a special
|
||||
// case — and a zero-part credit reaching here would otherwise
|
||||
// render as nothing at all, which is worse than the single-artist
|
||||
// answer it replaced.
|
||||
if (!parts || parts.length < 2) {
|
||||
return artistLink(fallbackName, fallbackMbid);
|
||||
}
|
||||
|
||||
return html`${parts.map(
|
||||
(part) =>
|
||||
html`${artistLink(part.creditedName, part.artistMbid)}${part.joinPhrase}`,
|
||||
)}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* The plain-text form of a credit, for `title=` attributes and any
|
||||
* other place that needs a string rather than a template.
|
||||
*
|
||||
* Rendered from the same parts by the same concatenation, so the
|
||||
* tooltip cannot disagree with the links beneath it.
|
||||
*/
|
||||
export function creditText(
|
||||
parts: readonly CreditPart[] | undefined,
|
||||
fallbackName: string,
|
||||
): string {
|
||||
if (!parts || parts.length < 2) return fallbackName;
|
||||
|
||||
return parts.map((p) => p.creditedName + p.joinPhrase).join('');
|
||||
}
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
/**
|
||||
* A track credited to more than one artist has one navigable artist in
|
||||
* this app and the rest are punctuation. `creditLink` is the fix: it
|
||||
* renders a credit as one link per credited artist with the join
|
||||
* phrases as plain text between them.
|
||||
*
|
||||
* The rule these tests exist to pin is that join phrases are
|
||||
* **assembly** instructions, not disassembly instructions — the credit
|
||||
* is built from its parts, never found by searching a name inside a
|
||||
* credit string. Measured on a real library, the stored credit text and
|
||||
* the catalog's parts disagree for about one in three multi-artist
|
||||
* credits, so a search would miss or match the wrong span.
|
||||
*/
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { html, render } from 'lit';
|
||||
|
||||
import { creditLink, creditText, type CreditPart } from '@utils/explore-link';
|
||||
|
||||
const TUPAC = '11111111-1111-4111-8111-111111111111';
|
||||
const SNOOP = '22222222-2222-4222-8222-222222222222';
|
||||
|
||||
const parts: CreditPart[] = [
|
||||
{ creditedName: '2Pac', artistMbid: TUPAC, joinPhrase: ' feat. ' },
|
||||
{ creditedName: 'Snoop Dogg', artistMbid: SNOOP, joinPhrase: '' },
|
||||
];
|
||||
|
||||
function renderToEl(value: unknown): HTMLElement {
|
||||
const host = document.createElement('div');
|
||||
render(html`${value}`, host);
|
||||
|
||||
return host;
|
||||
}
|
||||
|
||||
describe('creditLink', () => {
|
||||
it('renders one link per credited artist', () => {
|
||||
const el = renderToEl(creditLink(parts, '2Pac feat. Snoop Dogg', TUPAC));
|
||||
const links = el.querySelectorAll('a.explore-link');
|
||||
|
||||
expect(links).toHaveLength(2);
|
||||
expect(links[0]?.textContent).toBe('2Pac');
|
||||
expect(links[1]?.textContent).toBe('Snoop Dogg');
|
||||
});
|
||||
|
||||
it('puts the join phrase between the links as plain text', () => {
|
||||
const el = renderToEl(creditLink(parts, '2Pac feat. Snoop Dogg', TUPAC));
|
||||
|
||||
// The whole credit reads correctly...
|
||||
expect(el.textContent?.replace(/\s+/g, ' ').trim()).toBe(
|
||||
'2Pac feat. Snoop Dogg',
|
||||
);
|
||||
|
||||
// ...and " feat. " is not inside either link, which is the
|
||||
// difference between a credit and a link with punctuation in it.
|
||||
for (const link of el.querySelectorAll('a.explore-link')) {
|
||||
expect(link.textContent).not.toMatch(/feat/);
|
||||
}
|
||||
});
|
||||
|
||||
it('falls back to a single link when there are no parts', () => {
|
||||
const el = renderToEl(creditLink(undefined, 'Alina Baraz & Galimatias', TUPAC));
|
||||
const links = el.querySelectorAll('a.explore-link');
|
||||
|
||||
expect(links).toHaveLength(1);
|
||||
expect(links[0]?.textContent).toBe('Alina Baraz & Galimatias');
|
||||
});
|
||||
|
||||
it('does not split the fallback string on its separators', () => {
|
||||
// "&" and "with" appear inside real artist names — "Simon &
|
||||
// Garfunkel" is one artist — so a credit with no parts is one
|
||||
// link, always. This is the whole reason primaryArtist() does
|
||||
// not split on them either.
|
||||
const el = renderToEl(creditLink(undefined, 'Simon & Garfunkel', TUPAC));
|
||||
|
||||
expect(el.querySelectorAll('a.explore-link')).toHaveLength(1);
|
||||
});
|
||||
|
||||
it('treats a one-part credit as the single-link case', () => {
|
||||
// A zero- or one-part credit reaching the multi-artist branch
|
||||
// would render as nothing, or as a link with a dangling join
|
||||
// phrase after it.
|
||||
const one: CreditPart[] = [
|
||||
{ creditedName: 'Solo', artistMbid: TUPAC, joinPhrase: '' },
|
||||
];
|
||||
const el = renderToEl(creditLink(one, 'Solo', TUPAC));
|
||||
|
||||
expect(el.querySelectorAll('a.explore-link')).toHaveLength(1);
|
||||
expect(el.textContent?.trim()).toBe('Solo');
|
||||
});
|
||||
|
||||
it('renders the credited name, not the artist name', () => {
|
||||
// MusicBrainz credits "Snoop Dogg" on a track by the artist
|
||||
// called "Snoop Doggy Dogg". Display follows the credit;
|
||||
// navigation follows the MBID.
|
||||
const el = renderToEl(creditLink(parts, 'anything', TUPAC));
|
||||
|
||||
expect(el.textContent).toContain('Snoop Dogg');
|
||||
expect(el.textContent).not.toContain('Snoop Doggy Dogg');
|
||||
});
|
||||
});
|
||||
|
||||
describe('creditText', () => {
|
||||
it('reassembles the credit as a string', () => {
|
||||
expect(creditText(parts, 'ignored')).toBe('2Pac feat. Snoop Dogg');
|
||||
});
|
||||
|
||||
it('is the fallback string when there are no parts', () => {
|
||||
expect(creditText(undefined, 'Alina Baraz & Galimatias')).toBe(
|
||||
'Alina Baraz & Galimatias',
|
||||
);
|
||||
});
|
||||
|
||||
it('agrees with what creditLink renders', () => {
|
||||
// The tooltip and the links come from the same parts by the same
|
||||
// concatenation, so they cannot disagree.
|
||||
const el = renderToEl(creditLink(parts, 'ignored', TUPAC));
|
||||
|
||||
expect(el.textContent?.replace(/\s+/g, ' ').trim()).toBe(
|
||||
creditText(parts, 'ignored'),
|
||||
);
|
||||
});
|
||||
});
|
||||
+5
-1
@@ -26,7 +26,11 @@ pre-commit:
|
||||
go generate ./...
|
||||
if [ -n "$(git diff --name-only)" ]; then
|
||||
echo "Generated code is out of date. Run 'make generate' and stage the changes."
|
||||
git diff --stat
|
||||
# --no-pager, or this blocks forever on `less` waiting for a
|
||||
# keypress that a hook run without a tty will never get: the
|
||||
# commit hangs at exactly the moment it is trying to tell you
|
||||
# why it failed.
|
||||
git --no-pager diff --stat
|
||||
exit 1
|
||||
fi
|
||||
|
||||
|
||||
Executable
+89
@@ -0,0 +1,89 @@
|
||||
#!/bin/sh
|
||||
# Snapshot the index build's database, which is the only copy of it.
|
||||
#
|
||||
# `/srv/yellowjacket/index-cache` is the `YJ_HOME` the index job keeps
|
||||
# between runs (`.gitea/workflows/index-artifact.yml` mounts it at
|
||||
# `/cache`). Its catalog is *derived*, not downloaded: the only way to
|
||||
# rebuild it is to re-stream the MetaBrainz dumps, which is hours at a
|
||||
# rate that is someone else's to decide. On 2026-08-17 a schema repair
|
||||
# dropped it and cost exactly that.
|
||||
#
|
||||
# So it gets a snapshot, and this is the script a cron on that host runs.
|
||||
# It is deliberately not part of the workflow: a backup that only exists
|
||||
# while the thing it protects is being modified is not a backup.
|
||||
#
|
||||
# Usage (on the Gitea host):
|
||||
#
|
||||
# scripts/index-cache-snapshot.sh [SOURCE_HOME] [DEST_DIR] [KEEP]
|
||||
#
|
||||
# SOURCE_HOME default /srv/yellowjacket/index-cache
|
||||
# DEST_DIR default /srv/yellowjacket/index-snapshots
|
||||
# KEEP how many to retain, default 2
|
||||
#
|
||||
# Suggested cron — daily, and nowhere near the Monday 04:00 build:
|
||||
#
|
||||
# 30 5 * * * /path/to/index-cache-snapshot.sh >> /var/log/yj-index-snapshot.log 2>&1
|
||||
#
|
||||
# Three things about it are load-bearing.
|
||||
#
|
||||
# **`VACUUM INTO`, not `cp`.** The database may be open, and a byte copy
|
||||
# of a live SQLite file is a corrupt file with a plausible size.
|
||||
# `VACUUM INTO` takes a read lock, writes a consistent compacted copy,
|
||||
# and is safe while the index job is running — it costs the snapshot's
|
||||
# own write, not the source's availability.
|
||||
#
|
||||
# **The staging directory is not copied.** `/cache/data/explore-staging`
|
||||
# is a resumable checkpoint of work in flight; it is large, it changes
|
||||
# constantly, and a build resumes without it. What cannot be re-derived
|
||||
# cheaply is the finished catalog, which is in the database.
|
||||
#
|
||||
# **A snapshot that is not verified is a belief.** Each one is opened
|
||||
# and asked for its catalog row count before the old ones are rotated
|
||||
# out, so a run that produced an unreadable file leaves the previous
|
||||
# good snapshot in place and fails loudly.
|
||||
set -eu
|
||||
|
||||
SOURCE_HOME="${1:-/srv/yellowjacket/index-cache}"
|
||||
DEST_DIR="${2:-/srv/yellowjacket/index-snapshots}"
|
||||
KEEP="${3:-2}"
|
||||
|
||||
DB="$SOURCE_HOME/data/yj.db"
|
||||
STAMP=$(date +%Y%m%d-%H%M%S)
|
||||
OUT="$DEST_DIR/yj-index-$STAMP.db"
|
||||
|
||||
die() { echo "index-snapshot: $*" >&2; exit 1; }
|
||||
|
||||
command -v sqlite3 >/dev/null 2>&1 || die "sqlite3 is not installed"
|
||||
[ -f "$DB" ] || die "no database at $DB (is SOURCE_HOME right?)"
|
||||
|
||||
mkdir -p "$DEST_DIR"
|
||||
|
||||
# Headroom: the copy is at most the size of the source, usually less
|
||||
# (VACUUM compacts). Refusing here beats a half-written snapshot.
|
||||
need_kb=$(du -k "$DB" | cut -f1)
|
||||
free_kb=$(df -Pk "$DEST_DIR" | awk 'NR == 2 { print $4 }')
|
||||
[ "$free_kb" -gt "$need_kb" ] || die "not enough space in $DEST_DIR (need ~${need_kb}K, have ${free_kb}K)"
|
||||
|
||||
# A failed snapshot must leave nothing behind. `VACUUM INTO` refuses an
|
||||
# existing file, so a partial one from a disk-full write would block
|
||||
# every later run -- and worse, rotation counts files by name, so it
|
||||
# would eventually be kept *instead of* a good one.
|
||||
cleanup() { [ -n "${KEPT:-}" ] || rm -f "$OUT"; }
|
||||
trap cleanup EXIT
|
||||
|
||||
echo "index-snapshot: $DB -> $OUT"
|
||||
sqlite3 "$DB" "VACUUM INTO '$OUT'" || die "VACUUM INTO failed"
|
||||
|
||||
rows=$(sqlite3 "$OUT" "SELECT count(*) FROM explore_index" 2>/dev/null) \
|
||||
|| die "snapshot is unreadable: keeping the previous ones"
|
||||
[ "${rows:-0}" -gt 0 ] || die "snapshot has an empty catalog: keeping the previous ones"
|
||||
|
||||
KEPT=1
|
||||
|
||||
echo "index-snapshot: ok, $rows catalog rows, $(du -h "$OUT" | cut -f1)"
|
||||
|
||||
# Rotate only after the new one has been verified.
|
||||
ls -1t "$DEST_DIR"/yj-index-*.db 2>/dev/null | tail -n +"$((KEEP + 1))" | while read -r old; do
|
||||
echo "index-snapshot: removing $old"
|
||||
rm -f "$old"
|
||||
done
|
||||
+71
-14
@@ -1,17 +1,48 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# Every command in .pi/ is a `make` target on purpose: the Makefile is
|
||||
# the source of truth for *how* to invoke something, and the skill only
|
||||
# decides *which* and *in what order*. This check keeps that honest —
|
||||
# a renamed or deleted target turns into a failing commit rather than
|
||||
# into an agent confidently running a command that no longer exists.
|
||||
# Every command in the agent-facing docs is a `make` target on purpose:
|
||||
# the Makefile is the source of truth for *how* to invoke something, and
|
||||
# the docs only decide *which* and *in what order*. This check keeps
|
||||
# that honest — a renamed or deleted target turns into a failing commit
|
||||
# rather than into an agent confidently running a command that no longer
|
||||
# exists.
|
||||
#
|
||||
# It extracts every `make <target>` mentioned under .pi/ and asserts the
|
||||
# target exists. Usage: scripts/skill-check.sh
|
||||
# It checks two things. Usage: scripts/skill-check.sh
|
||||
#
|
||||
# **Every `make <target>` named in an agent-facing doc exists.** The
|
||||
# scanned set is `.pi/` *and* CLAUDE.md, which is the half that was
|
||||
# missing: CLAUDE.md names 27 targets and nothing verified one of them,
|
||||
# so the file the agents trust most was the file least checked.
|
||||
#
|
||||
# **AGENTS.md is a symlink to CLAUDE.md.** This repo is worked on by
|
||||
# two agent harnesses that read different files by convention — Claude
|
||||
# Code reads CLAUDE.md, others read AGENTS.md — and two harnesses
|
||||
# reading two descriptions of one project is how they come to hold
|
||||
# different beliefs about it. A symlink makes that impossible by
|
||||
# construction; a *copy* would pass every other check in this repo while
|
||||
# silently drifting, which is exactly the failure being prevented, so
|
||||
# the symlink itself is asserted rather than its contents compared.
|
||||
set -euo pipefail
|
||||
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
# The symlink half runs even without .pi/, since it is not about .pi/.
|
||||
if [ -e AGENTS.md ] || [ -L AGENTS.md ]; then
|
||||
if [ ! -L AGENTS.md ]; then
|
||||
echo "skill-check: AGENTS.md is a regular file, not a symlink to CLAUDE.md." >&2
|
||||
echo " Two harnesses would read two descriptions of one project." >&2
|
||||
echo " Fix: rm AGENTS.md && ln -s CLAUDE.md AGENTS.md" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
target="$(readlink AGENTS.md)"
|
||||
|
||||
if [ "$target" != "CLAUDE.md" ]; then
|
||||
echo "skill-check: AGENTS.md points at '$target', expected CLAUDE.md." >&2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
[ -d .pi ] || exit 0
|
||||
|
||||
# `make -pq` prints the database including every rule, without running
|
||||
@@ -21,11 +52,37 @@ targets="$({ make -pqRr 2>/dev/null || true; } |
|
||||
awk '/^[a-zA-Z0-9][^$#\/\t=]*:([^=]|$)/ {sub(/:.*/, "", $0); print}' |
|
||||
sort -u)"
|
||||
|
||||
# A mention counts only when it is code: backticked (`make ui-test`) or
|
||||
# the first thing on a line, as in a fenced block. Bare prose is not
|
||||
# scanned, because English says things like "a renamed make target".
|
||||
mentioned="$(grep -rhoE '(`|^)make [a-z][a-z0-9-]*' .pi --include='*.md' |
|
||||
sed 's/^`//' | awk '{print $2}' | sort -u)"
|
||||
# A mention counts only when it is code: backticked (`make ui-test`)
|
||||
# anywhere, or at the start of a line **inside a fenced block**. Bare
|
||||
# prose is not scanned, because English says things like "a renamed make
|
||||
# target".
|
||||
#
|
||||
# The fence is why this is awk rather than one grep. Line-start alone is
|
||||
# not evidence of code in a file that is mostly hard-wrapped prose: the
|
||||
# sentence "Two green branches do not / make a green merge" wrapped onto
|
||||
# a line beginning `make a`, and the check duly failed on a target called
|
||||
# `a`. Inside a fence it is code; outside one it is a sentence that
|
||||
# happened to break there, and a check that fails on reflow gets
|
||||
# disabled rather than fixed.
|
||||
#
|
||||
# AGENTS.md is deliberately not in this list: it is a symlink to
|
||||
# CLAUDE.md, asserted above, so scanning it would report every failure
|
||||
# twice under two names.
|
||||
mentioned="$({ find .pi -name '*.md' 2>/dev/null; echo CLAUDE.md; } |
|
||||
xargs awk '
|
||||
FNR == 1 { fence = 0 }
|
||||
/^```/ { fence = !fence; next }
|
||||
{
|
||||
rest = $0
|
||||
while (match(rest, /`make [a-z][a-z0-9-]*/)) {
|
||||
print substr(rest, RSTART + 6, RLENGTH - 6)
|
||||
rest = substr(rest, RSTART + RLENGTH)
|
||||
}
|
||||
if (fence && match($0, /^make [a-z][a-z0-9-]*/)) {
|
||||
print substr($0, 6, RLENGTH - 5)
|
||||
}
|
||||
}
|
||||
' | sort -u)"
|
||||
|
||||
missing=""
|
||||
|
||||
@@ -36,10 +93,10 @@ for t in $mentioned; do
|
||||
done
|
||||
|
||||
if [ -n "$missing" ]; then
|
||||
echo "skill-check: .pi/ documents make targets that do not exist:" >&2
|
||||
echo "skill-check: the agent docs name make targets that do not exist:" >&2
|
||||
for t in $missing; do
|
||||
echo " make $t" >&2
|
||||
grep -rln "make $t" .pi --include='*.md' | sed 's/^/ /' >&2
|
||||
grep -rln "make $t" .pi CLAUDE.md --include='*.md' | sed 's/^/ /' >&2
|
||||
done
|
||||
echo "Fix the docs, or restore the target." >&2
|
||||
exit 1
|
||||
|
||||
Reference in New Issue
Block a user