test(database): the next destructive repair fails a test, not a volume
Build & publish Arch package / arch-package (push) Successful in 2m33s
CI / check (push) Successful in 3m14s
CI / e2e (push) Canceled after 3m3s

The fix for the dropped catalog pins one table in one wrong shape, which
is the failure that happened. What cost the rebuild was more general: a
destructive repair added at `database.NewDB` -- the chokepoint every
binary in this project shares -- without asking which binary it runs in.
The next one will have a different name and a different reason.

So `TestNoCacheTableIsRetiredHere` asserts the outcome instead: put every
`datamap` Cache table into a shape the schema has moved past, open the
database the way cmd/indexbuild does, and require all of them to still be
there. Driving it from `datamap.ByKind` is what makes it cover tables
nobody remembered -- flipping the policy back fails on five, including
the two artist-credit tables added the same day, where the existing test
fails on one. It asserts the rows survive too, because SQLite does an
implicit DELETE before a DROP and a repair that recreated the table would
look identical. And it accepts an error from `NewDB`, because that is the
documented trade: loud is recoverable, gone is not.

`scripts/index-cache-snapshot.sh` covers the half no test can reach. The
volume holds the only copy of a catalog that costs hours of someone
else's bandwidth to re-derive. `VACUUM INTO` rather than `cp`, since a
byte copy of a live SQLite file is a corrupt file of plausible size; the
resumable staging directory is skipped; and each snapshot is reopened and
asked for its catalog row count before anything is rotated out. A corrupt
source and an empty catalog were both exercised: each exits non-zero,
removes its own output, and leaves the previous snapshots alone.

docs/index-cache.md is the restore, and the reason to bother: a restored
snapshot resolves to `refresh` and folds in the listens since, which is
minutes against the 3-23h this rebuild has been estimating.
This commit is contained in:
2026-08-17 13:56:08 -04:00
parent 8c48105ca3
commit c03c0b8ec4
5 changed files with 328 additions and 0 deletions
+89
View File
@@ -0,0 +1,89 @@
#!/bin/sh
# Snapshot the index build's database, which is the only copy of it.
#
# `/srv/yellowjacket/index-cache` is the `YJ_HOME` the index job keeps
# between runs (`.gitea/workflows/index-artifact.yml` mounts it at
# `/cache`). Its catalog is *derived*, not downloaded: the only way to
# rebuild it is to re-stream the MetaBrainz dumps, which is hours at a
# rate that is someone else's to decide. On 2026-08-17 a schema repair
# dropped it and cost exactly that.
#
# So it gets a snapshot, and this is the script a cron on that host runs.
# It is deliberately not part of the workflow: a backup that only exists
# while the thing it protects is being modified is not a backup.
#
# Usage (on the Gitea host):
#
# scripts/index-cache-snapshot.sh [SOURCE_HOME] [DEST_DIR] [KEEP]
#
# SOURCE_HOME default /srv/yellowjacket/index-cache
# DEST_DIR default /srv/yellowjacket/index-snapshots
# KEEP how many to retain, default 2
#
# Suggested cron — daily, and nowhere near the Monday 04:00 build:
#
# 30 5 * * * /path/to/index-cache-snapshot.sh >> /var/log/yj-index-snapshot.log 2>&1
#
# Three things about it are load-bearing.
#
# **`VACUUM INTO`, not `cp`.** The database may be open, and a byte copy
# of a live SQLite file is a corrupt file with a plausible size.
# `VACUUM INTO` takes a read lock, writes a consistent compacted copy,
# and is safe while the index job is running — it costs the snapshot's
# own write, not the source's availability.
#
# **The staging directory is not copied.** `/cache/data/explore-staging`
# is a resumable checkpoint of work in flight; it is large, it changes
# constantly, and a build resumes without it. What cannot be re-derived
# cheaply is the finished catalog, which is in the database.
#
# **A snapshot that is not verified is a belief.** Each one is opened
# and asked for its catalog row count before the old ones are rotated
# out, so a run that produced an unreadable file leaves the previous
# good snapshot in place and fails loudly.
set -eu
SOURCE_HOME="${1:-/srv/yellowjacket/index-cache}"
DEST_DIR="${2:-/srv/yellowjacket/index-snapshots}"
KEEP="${3:-2}"
DB="$SOURCE_HOME/data/yj.db"
STAMP=$(date +%Y%m%d-%H%M%S)
OUT="$DEST_DIR/yj-index-$STAMP.db"
die() { echo "index-snapshot: $*" >&2; exit 1; }
command -v sqlite3 >/dev/null 2>&1 || die "sqlite3 is not installed"
[ -f "$DB" ] || die "no database at $DB (is SOURCE_HOME right?)"
mkdir -p "$DEST_DIR"
# Headroom: the copy is at most the size of the source, usually less
# (VACUUM compacts). Refusing here beats a half-written snapshot.
need_kb=$(du -k "$DB" | cut -f1)
free_kb=$(df -Pk "$DEST_DIR" | awk 'NR == 2 { print $4 }')
[ "$free_kb" -gt "$need_kb" ] || die "not enough space in $DEST_DIR (need ~${need_kb}K, have ${free_kb}K)"
# A failed snapshot must leave nothing behind. `VACUUM INTO` refuses an
# existing file, so a partial one from a disk-full write would block
# every later run -- and worse, rotation counts files by name, so it
# would eventually be kept *instead of* a good one.
cleanup() { [ -n "${KEPT:-}" ] || rm -f "$OUT"; }
trap cleanup EXIT
echo "index-snapshot: $DB -> $OUT"
sqlite3 "$DB" "VACUUM INTO '$OUT'" || die "VACUUM INTO failed"
rows=$(sqlite3 "$OUT" "SELECT count(*) FROM explore_index" 2>/dev/null) \
|| die "snapshot is unreadable: keeping the previous ones"
[ "${rows:-0}" -gt 0 ] || die "snapshot has an empty catalog: keeping the previous ones"
KEPT=1
echo "index-snapshot: ok, $rows catalog rows, $(du -h "$OUT" | cut -f1)"
# Rotate only after the new one has been verified.
ls -1t "$DEST_DIR"/yj-index-*.db 2>/dev/null | tail -n +"$((KEEP + 1))" | while read -r old; do
echo "index-snapshot: removing $old"
rm -f "$old"
done