docs/issue-driven-workflow
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 logan
|
c03c0b8ec4 |
test(database): the next destructive repair fails a test, not a volume
The fix for the dropped catalog pins one table in one wrong shape, which is the failure that happened. What cost the rebuild was more general: a destructive repair added at `database.NewDB` -- the chokepoint every binary in this project shares -- without asking which binary it runs in. The next one will have a different name and a different reason. So `TestNoCacheTableIsRetiredHere` asserts the outcome instead: put every `datamap` Cache table into a shape the schema has moved past, open the database the way cmd/indexbuild does, and require all of them to still be there. Driving it from `datamap.ByKind` is what makes it cover tables nobody remembered -- flipping the policy back fails on five, including the two artist-credit tables added the same day, where the existing test fails on one. It asserts the rows survive too, because SQLite does an implicit DELETE before a DROP and a repair that recreated the table would look identical. And it accepts an error from `NewDB`, because that is the documented trade: loud is recoverable, gone is not. `scripts/index-cache-snapshot.sh` covers the half no test can reach. The volume holds the only copy of a catalog that costs hours of someone else's bandwidth to re-derive. `VACUUM INTO` rather than `cp`, since a byte copy of a live SQLite file is a corrupt file of plausible size; the resumable staging directory is skipped; and each snapshot is reopened and asked for its catalog row count before anything is rotated out. A corrupt source and an empty catalog were both exercised: each exits non-zero, removes its own output, and leaves the previous snapshots alone. docs/index-cache.md is the restore, and the reason to bother: a restored snapshot resolves to `refresh` and folds in the listens since, which is minutes against the 3-23h this rebuild has been estimating. |
||
  yonluandClaude Opus 5
|
4f8257ef72 |
fix(database): never retire the catalog the index build derives
The stale-shape repair dropped the CI catalog on its first run:
retiring a table ... table=explore_index
reason="column entity_type is TEXT, schema declares INTEGER"
index maintenance mode=build reason="no completed import yet"
The mismatch was real and the drop was correct by the app's rule: a
client's catalog is *downloaded*, so a wrong shape costs a minute of
re-fetching the artifact, while keeping it costs every Explore read.
It is the wrong rule for one database. cmd/indexbuild's catalog is not
downloaded, it is what the artifact is cut from — the only way back is
the ~205 GB dump stream the /cache volume exists to avoid. And that
database is deliberately kept in the older encoding, which
`fix(indexexport): read an index older than the binary` exists to
tolerate, so the shape does not match by design and would have been
dropped on every run.
retireLibraryTables, right beside it, never touches the catalog for
exactly this reason. The repair reached past that protection because it
runs inside database.NewDB, which cmd/indexbuild also calls.
So the policy is a build tag, which is how this project already tells
the index tools apart (runtime_indexbuild.go, servicestartup.go,
dumpbuild_stub.go): Cache tables are rebuilt in the app and never in
cmd/indexbuild. Owned and Derived are still repaired in both — that is
the half this database can safely discard, and retireLibraryTables
already discards it.
The residual trade is deliberate: a future explore_index column will
now fail the index job loudly on applySchema rather than silently
costing it a 205 GB rebuild. A human should decide that one.
TestTheCatalogSurvivesAStaleShape is the accident, symptom first, with
the shape the real database is in — every current column, ids and
entity type still text. It fails with "the catalog was retired" when
the policy is flipped back.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AfVYUVExXsx1nSWrXN8mAh
|
||
|
yonluandClaude Opus 5
|
eb059a3d71 |
fix(database): retire a table whose shape the schema moved past
`applySchema` is CREATE ... IF NOT EXISTS and there is no migration chain, so a *changed* table never migrates: the statement silently no-ops against the old shape. Two plans had already landed on that, and neither showed up in a test because a fresh install is perfectly healthy. - 014 added `total_tracks` to explore_index and to `indexRowFields`, the projection every explore read uses, so every search, browse, artist page and album page failed with "no such column: total_tracks" on any database that already had a catalog. - 013 reshaped audio_files, so applySchema could not run at all and the app did not open. staleshape.go runs before applySchema and drops what disagrees, so the create is a create. It parses sql/schemas/ for the expectation rather than writing the column list down a second time, and it notices a changed *type* as well as a missing column — 013 moved mbid TEXT to BLOB, which no ALTER could express and which SQLite will not coerce, so a query against 16 raw bytes returns no rows rather than an error. Only Authored tables are exempt. Cache is rebuildable by definition, Owned is what a rescan rebuilds (plan 013's stated "delete and rescan"), and a table the schema no longer describes at all goes too -- 013 left seven behind plus schema_migrations. Three things in it are load-bearing, and each was a bug first: - The parser read `UNIQUE(mbid)` as a column, which made a healthy catalog look stale. That would have retired it on every launch and cost every user an artifact download per start. - The drops are one transaction with defer_foreign_keys. Those legacy tables reference each other, so any order fails on whichever goes first; turning foreign keys off instead would suppress playlist_tracks.audio_file_id's ON DELETE SET NULL and leave entries pointing at ids a rescan reissues to *different songs*. Nulled entries are empty; stale ones are wrong, and wrong quietly. - The order is sorted, so a failure reproduces. Map order is random, and the foreign-key bug passed its own regression test on two runs in three until the order was fixed. Verified against a real pre-013 install: it opens, its 22 playlists survive, 1,887 linked playlist entries become 0 rather than dangling, and the legacy tables are swept. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AfVYUVExXsx1nSWrXN8mAh |
||
|
logan
|
66182f82cd |
fix(indexbuild): repair the one database a squash cannot reach
The index job's /cache volume is a real YJ_HOME that outlives every run, so plan 013's reshaped audio_files met a database still in the old shape: `CREATE INDEX ... album_id` against a table without that column, on every launch. "Delete and rescan" is the squash's answer and is free everywhere except here, where half the file is the catalog and deleting it costs ~205GB of downloading. indexbuild now drops every table datamap does not classify as Cache before the schema is applied. Nothing scans, plays or authors in that database, so its non-catalog half is empty by construction and a shape the schema stopped describing is pure liability; the catalog is never touched. TestRetireLibraryTables reproduces the failure symptom-first: build the real schema, put audio_files back the way the volume had it, assert the open fails, then assert the repair makes it open with the catalog row still there. |