main
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 logan
|
c03c0b8ec4 |
test(database): the next destructive repair fails a test, not a volume
The fix for the dropped catalog pins one table in one wrong shape, which is the failure that happened. What cost the rebuild was more general: a destructive repair added at `database.NewDB` -- the chokepoint every binary in this project shares -- without asking which binary it runs in. The next one will have a different name and a different reason. So `TestNoCacheTableIsRetiredHere` asserts the outcome instead: put every `datamap` Cache table into a shape the schema has moved past, open the database the way cmd/indexbuild does, and require all of them to still be there. Driving it from `datamap.ByKind` is what makes it cover tables nobody remembered -- flipping the policy back fails on five, including the two artist-credit tables added the same day, where the existing test fails on one. It asserts the rows survive too, because SQLite does an implicit DELETE before a DROP and a repair that recreated the table would look identical. And it accepts an error from `NewDB`, because that is the documented trade: loud is recoverable, gone is not. `scripts/index-cache-snapshot.sh` covers the half no test can reach. The volume holds the only copy of a catalog that costs hours of someone else's bandwidth to re-derive. `VACUUM INTO` rather than `cp`, since a byte copy of a live SQLite file is a corrupt file of plausible size; the resumable staging directory is skipped; and each snapshot is reopened and asked for its catalog row count before anything is rotated out. A corrupt source and an empty catalog were both exercised: each exits non-zero, removes its own output, and leaves the previous snapshots alone. docs/index-cache.md is the restore, and the reason to bother: a restored snapshot resolves to `refresh` and folds in the listens since, which is minutes against the 3-23h this rebuild has been estimating. |
||
  yonluandClaude Opus 5
|
4f8257ef72 |
fix(database): never retire the catalog the index build derives
The stale-shape repair dropped the CI catalog on its first run:
retiring a table ... table=explore_index
reason="column entity_type is TEXT, schema declares INTEGER"
index maintenance mode=build reason="no completed import yet"
The mismatch was real and the drop was correct by the app's rule: a
client's catalog is *downloaded*, so a wrong shape costs a minute of
re-fetching the artifact, while keeping it costs every Explore read.
It is the wrong rule for one database. cmd/indexbuild's catalog is not
downloaded, it is what the artifact is cut from — the only way back is
the ~205 GB dump stream the /cache volume exists to avoid. And that
database is deliberately kept in the older encoding, which
`fix(indexexport): read an index older than the binary` exists to
tolerate, so the shape does not match by design and would have been
dropped on every run.
retireLibraryTables, right beside it, never touches the catalog for
exactly this reason. The repair reached past that protection because it
runs inside database.NewDB, which cmd/indexbuild also calls.
So the policy is a build tag, which is how this project already tells
the index tools apart (runtime_indexbuild.go, servicestartup.go,
dumpbuild_stub.go): Cache tables are rebuilt in the app and never in
cmd/indexbuild. Owned and Derived are still repaired in both — that is
the half this database can safely discard, and retireLibraryTables
already discards it.
The residual trade is deliberate: a future explore_index column will
now fail the index job loudly on applySchema rather than silently
costing it a 205 GB rebuild. A human should decide that one.
TestTheCatalogSurvivesAStaleShape is the accident, symptom first, with
the shape the real database is in — every current column, ids and
entity type still text. It fails with "the catalog was retired" when
the policy is flipped back.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AfVYUVExXsx1nSWrXN8mAh
|
||
|
yonluandClaude Opus 5
|
eb059a3d71 |
fix(database): retire a table whose shape the schema moved past
`applySchema` is CREATE ... IF NOT EXISTS and there is no migration chain, so a *changed* table never migrates: the statement silently no-ops against the old shape. Two plans had already landed on that, and neither showed up in a test because a fresh install is perfectly healthy. - 014 added `total_tracks` to explore_index and to `indexRowFields`, the projection every explore read uses, so every search, browse, artist page and album page failed with "no such column: total_tracks" on any database that already had a catalog. - 013 reshaped audio_files, so applySchema could not run at all and the app did not open. staleshape.go runs before applySchema and drops what disagrees, so the create is a create. It parses sql/schemas/ for the expectation rather than writing the column list down a second time, and it notices a changed *type* as well as a missing column — 013 moved mbid TEXT to BLOB, which no ALTER could express and which SQLite will not coerce, so a query against 16 raw bytes returns no rows rather than an error. Only Authored tables are exempt. Cache is rebuildable by definition, Owned is what a rescan rebuilds (plan 013's stated "delete and rescan"), and a table the schema no longer describes at all goes too -- 013 left seven behind plus schema_migrations. Three things in it are load-bearing, and each was a bug first: - The parser read `UNIQUE(mbid)` as a column, which made a healthy catalog look stale. That would have retired it on every launch and cost every user an artifact download per start. - The drops are one transaction with defer_foreign_keys. Those legacy tables reference each other, so any order fails on whichever goes first; turning foreign keys off instead would suppress playlist_tracks.audio_file_id's ON DELETE SET NULL and leave entries pointing at ids a rescan reissues to *different songs*. Nulled entries are empty; stale ones are wrong, and wrong quietly. - The order is sorted, so a failure reproduces. Map order is random, and the foreign-key bug passed its own regression test on two runs in three until the order was fixed. Verified against a real pre-013 install: it opens, its 22 playlists survive, 1,887 linked playlist entries become 0 rather than dangling, and the legacy tables are swept. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AfVYUVExXsx1nSWrXN8mAh |
||
|
yonluandClaude Opus 5
|
b3737d30af |
feat(explore): carry multi-artist credits in the catalog
A track credited to more than one artist has exactly one navigable artist in this app and the rest are punctuation. `primaryArtist()` string-parses the credit, strips a " feat. " clause and discards the guest; it deliberately does not split on "&", "with" or "," because those live inside real artist names. Measured on a real 26,069-file library plus an 80+80 MusicBrainz sample: 13% of recordings are multi-artist upstream, while only 0.86% of files carry any structured multi-artist tag — mp3 carries zero files with multiple MUSICBRAINZ_ARTISTID across 19,840. Of 1,286 files saying "feat.", 90% have nothing structured behind it, and a sample of 80 such files was multi-artist in MB 80 times out of 80. CLAUDE.md justified plan 013's removal of the credit tables with "3 credits of 2,823 listed more than one artist". That measured our own *writer* — cachedLinkArtist was called once per credit, so a collaboration could never have been recorded. Dropping the join table was still right on cost; the evidence for "multi-artist is rare" was not. A credit is ordered parts and the credit string is derived from them, so join phrases are assembly instructions, not disassembly ones. Nothing here reconstructs a credit by searching a name inside a credit string: the stored text may come from tags while the parts come from the catalog, and those disagree for ~1 in 3 multi-artist credits. Where it comes from, after two dead ends: the canonical dump CI already streams has no join phrases and no as-credited names, and the JSON dumps cover 153,691 recordings of ~35M with *zero* overlap against a real library. So mbdump.tar.bz2 — 7.1 GB, ~13.7 min in pure-Go bzip2, whose members are alphabetical, which is what lets one pass resolve an entity's credit without buffering 35M recordings. - artist_credit_part / artist_credit_ref, multi-artist credits only: a single-artist credit is already explore_index's own artist_name. - Column layouts verified against the real 20260815 export; ErrDumpShape makes a wrong guess a failed build, not a wrong catalog. - The pass runs on every mode, not just a build. The job picks its mode from the index's own state, and a complete import means "refresh", which never enters the importer — so credits could otherwise only arrive via a rebuild that re-downloads ~205 GB. It reports whether it populated anything, which is what flips `changed` and republishes. - The importer asks whether an artifact carries the tables, on the writer where `core` is attached, so the artifact already published still imports. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AfVYUVExXsx1nSWrXN8mAh |
||
|
logan
|
66182f82cd |
fix(indexbuild): repair the one database a squash cannot reach
The index job's /cache volume is a real YJ_HOME that outlives every run, so plan 013's reshaped audio_files met a database still in the old shape: `CREATE INDEX ... album_id` against a table without that column, on every launch. "Delete and rescan" is the squash's answer and is free everywhere except here, where half the file is the catalog and deleting it costs ~205GB of downloading. indexbuild now drops every table datamap does not classify as Cache before the schema is applied. Nothing scans, plays or authors in that database, so its non-catalog half is empty by construction and a shape the schema stopped describing is pure liability; the catalog is never touched. TestRetireLibraryTables reproduces the failure symptom-first: build the real schema, put audio_files back the way the volume had it, assert the open fails, then assert the repair makes it open with the catalog row still there. |
||
|
logan
|
b98840ee37 |
fix(build): keep the index tools free of the Wails application
The v3 migration put application.Get() in backend/events and a ServiceStartup hook in backend/explore, both of which cmd/indexbuild reaches. v3's application package is GTK/WebKit bindings on Linux, so the index-artifact job — a plain golang container with CGO_ENABLED=0, on the stated grounds that neither command imports the app — stopped compiling with "undefined: pointer". That job owns the ~205 GB dump checkpoint, so it is the worst place to learn this. Both are behind the indexbuild tag now: the one app.Event.Emit lives in runtime_wails.go, runtime_indexbuild.go answers ErrNoRuntime (what the app itself returns before Run, so Deliver's callers need no second path), and explore's ServiceStartup moves to its own tagged file. TestIndexToolsDoNotImportWails walks `go list -deps -tags indexbuild` so the claim the workflow makes is checked rather than assumed. |
||
|
yonluandClaude Sonnet 5
|
e190fd75b9 |
feat: data lifecycle rewrite, download clients, wanted list, and central catalog index
Ships the fresh-start schema cleanup: rebuilt explore catalog index pipeline (dump import, artifact fetch/build, incremental listen-count refresh), a new download subsystem (Lidarr/Prowlarr/qBittorrent/SABnzbd/ slskd/yt-dlp providers, staging, reconciliation, wanted list), and the supporting schema/query/store changes across backend and frontend. Also includes two smaller follow-ups: bump the central index's rebuild-after cadence from 90 to 180 days, and remove the Explore "library only" online/offline toggle entirely (frontend-only, no backend counterpart) rather than carry unused UI/state. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y2Agd9af5hE7qzti2ackiS |
||
|
yonluandClaude Opus 5
|
01bc5f2094 |
feat(jobs): surface background jobs with progress, logs and controls
Add a central job registry that library scans and search index builds report into, so background work is visible instead of buried in the settings page. - backend/jobs: registry with per-job ring-buffer logs, capability-driven controls, and one coalesced JobsChanged snapshot at 4Hz - pause survives restart via a job_state table; a paused scan is adopted back on launch and skipped by the soft scan - top-bar indicator, popover, details drawer and a Jobs page replacing the config page's scan UI; per-library start/stop retained - scan timing breakdown moves into the job log, Full rescan to the Jobs page; delete the orphaned library-manager component Also add cmd/indexbuild and cmd/indexexport so the explore index can be built once centrally rather than by every install, which today streams ~205GB from the ListenBrainz spark dump on first run. indexbuild picks build/refresh/rebuild from index state; the Gitea workflow runs it on push, weekly, or manually and publishes only when content changed. fresh-install no longer defaults YJ_HOME under /tmp: it is tmpfs on most distros, and the import needs ~6GB of real disk. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |