yonlu
5d9c677cf7
Nothing should be published until the code that imports it on a user's machine has imported it here. The exporter and the importer are two descriptions of one storage format, and every other tier tests the importer against a *fixture* rather than against the file being shipped — a second description free to be wrong in the same direction as the code reading it. That is how #258 reached everyone: the importer positioned its batch walk with a Go `string` cursor against this file's 16-byte `mbid` column, and SQLite neither coerces between TEXT and BLOB nor complains about the comparison, so the walk merged no rows and never advanced. The fixture guarding that walk writes the old text encoding, and the only compact fixture is one row, below the batch size, so the bound query never ran. Both were green throughout, and no install could finish a first index build. `TestImportPublishedArtifact` takes the published file and runs the client's own path over it — checksum, decompress, merge — and asserts that what the artifact holds is what the client ends up with: the row count, the rows carrying a listen count, an FTS index in step with the table, and one row read back through the app's own MBID conversions. It skips without `YJ_CORE_INDEX_ARTIFACT`, so an ordinary run pays nothing. It needs no Wails, which is why the step runs it under the indexbuild tag: that container has no GTK. Measured on the current artifact, 64.8 MB compressed: 39 seconds including the decompress. Verified against the pre-fix comparison behaviour, the step goes red in about 3 seconds — the strictly-advancing guard fails the import with a named reason rather than the day-long spin it used to produce. Refs #258
231 lines
10 KiB
YAML
231 lines
10 KiB
YAML
name: Search index maintenance
|
|
|
|
# indexbuild decides what to do from the index's own state, so every
|
|
# trigger below runs the same command:
|
|
#
|
|
# no completed import -> build (first run, or resume a partial one)
|
|
# import older than 6mo -> rebuild (re-import from the newest dump)
|
|
# otherwise -> refresh (fold in new incremental listens)
|
|
#
|
|
# **There is deliberately no `push` trigger, and restoring one is a
|
|
# decision rather than a cleanup.** A refresh is individually cheap, so
|
|
# running it on every push to main looked free; what it actually does is
|
|
# put an unattended job that mutates the only copy of a ~205 GB catalog
|
|
# on the same trigger as an ordinary code change, on a runner with
|
|
# capacity 1.
|
|
#
|
|
# That is not hypothetical. On 2026-08-17 `fix(database): retire a table
|
|
# whose shape the schema moved past` landed on main, green — the CI
|
|
# database is deliberately in the older encoding, so the stale-shape
|
|
# repair judged its `explore_index` stale and dropped it, and this job
|
|
# fell back to a full import from the dumps. `fix(database): never
|
|
# retire the catalog the index build derives` stops that specific repair
|
|
# and cannot undo it. Every push to main then booked another `budget`
|
|
# (3h) of the one runner while ordinary CI queued behind it.
|
|
#
|
|
# So the rule this file is an instance of: **a job that mutates state
|
|
# which cannot be rebuilt in ten minutes is triggered deliberately, not
|
|
# by a push.** The weekly cron keeps the catalog current, and
|
|
# workflow_dispatch resumes or forces a build — indexbuild picks up from
|
|
# its checkpoint either way, so nothing is lost by not running on every
|
|
# merge. See docs/index-cache.md for the snapshot and the restore.
|
|
on:
|
|
schedule:
|
|
# Weekly update pass. The 6-month rebuild is triggered by the same
|
|
# command when it notices the import has aged out.
|
|
- cron: '0 4 * * 1'
|
|
workflow_dispatch:
|
|
inputs:
|
|
mode:
|
|
description: 'auto | build | refresh | rebuild'
|
|
required: false
|
|
default: 'auto'
|
|
budget:
|
|
description: 'Max build time this run'
|
|
required: false
|
|
default: '3h'
|
|
artists:
|
|
description: 'Top artists in the core artifact'
|
|
required: false
|
|
default: '50000'
|
|
|
|
# Runs share one persistent working directory, so they must not overlap.
|
|
# A push landing mid-build waits rather than corrupting the checkpoint.
|
|
#
|
|
# That directory holds the only copy of a catalog nothing can cheaply
|
|
# re-derive: see docs/index-cache.md for the snapshot it takes and the
|
|
# restore, which is minutes against the hours a rebuild costs.
|
|
concurrency:
|
|
group: search-index
|
|
cancel-in-progress: false
|
|
|
|
jobs:
|
|
maintain-index:
|
|
runs-on: ubuntu-latest
|
|
container:
|
|
# CGO is not needed: the project uses the pure-Go modernc sqlite
|
|
# driver, and neither command imports the Wails app — which is a
|
|
# claim with a test behind it now (cmd/indexbuild/deps_test.go),
|
|
# because the v3 migration quietly broke it and this job was where
|
|
# that surfaced.
|
|
image: golang:1.26
|
|
# This host path must exist on the runner and be listed verbatim in
|
|
# act_runner's container.valid_volumes. It holds explore-staging/
|
|
# (counts.bin + state.json) and yj.db — the checkpoint that makes
|
|
# resuming possible. Losing it means re-downloading ~205GB.
|
|
volumes:
|
|
- /srv/yellowjacket/index-cache:/cache
|
|
env:
|
|
YJ_HOME: /cache
|
|
CGO_ENABLED: '0'
|
|
PACKAGE_TOKEN: ${{ secrets.PACKAGE_TOKEN }}
|
|
SERVER_URL: ${{ github.server_url }}
|
|
OWNER: ${{ github.repository_owner }}
|
|
REPO: ${{ github.repository }}
|
|
SHA: ${{ github.sha }}
|
|
MODE: ${{ inputs.mode || 'auto' }}
|
|
BUDGET: ${{ inputs.budget || '3h' }}
|
|
ARTISTS: ${{ inputs.artists || '50000' }}
|
|
steps:
|
|
# Cloned by hand rather than with actions/checkout: that is a JS
|
|
# action and needs node inside the job container, which the golang
|
|
# image does not carry. Same approach as arch-package.yml.
|
|
- name: Clone repo at this commit
|
|
run: |
|
|
set -eu
|
|
git clone --quiet \
|
|
"https://x-access-token:${PACKAGE_TOKEN}@${SERVER_URL#https://}/${REPO}.git" \
|
|
/src
|
|
git -C /src checkout --quiet --detach "$SHA"
|
|
git -C /src log --oneline -1
|
|
|
|
- name: Verify the cache volume
|
|
run: |
|
|
set -eu
|
|
mkdir -p /cache
|
|
# A RAM-backed cache would defeat the point: the checkpoint has
|
|
# to outlive the job, and the import wants real disk headroom.
|
|
fstype=$(stat -f -c %T /cache || echo unknown)
|
|
echo "cache fstype: $fstype"
|
|
case "$fstype" in
|
|
tmpfs|ramfs)
|
|
echo "::error::/cache is RAM-backed; use a disk-backed host path."
|
|
exit 1 ;;
|
|
esac
|
|
df -h /cache
|
|
|
|
- name: Build tools
|
|
working-directory: /src
|
|
# The dump importer is behind the `indexbuild` tag so it is not
|
|
# linked into the app binary; cmd/indexbuild carries the same tag
|
|
# and will not build without it.
|
|
run: |
|
|
go build -tags indexbuild -o /usr/local/bin/ ./cmd/indexbuild
|
|
go build -o /usr/local/bin/ ./cmd/indexexport
|
|
|
|
- name: Maintain index
|
|
id: maintain
|
|
run: |
|
|
set +e
|
|
indexbuild -mode "$MODE" -budget "$BUDGET"
|
|
code=$?
|
|
set -e
|
|
case "$code" in
|
|
0) ;;
|
|
3) echo "::notice::Build checkpointed with work remaining — rerun to continue." ;;
|
|
*) exit "$code" ;;
|
|
esac
|
|
|
|
# Publishing only on `changed` keeps identical artifacts from
|
|
# accumulating when a refresh finds nothing new.
|
|
- name: Export core artifact
|
|
if: steps.maintain.outputs.complete == 'true' && steps.maintain.outputs.changed == 'true'
|
|
run: |
|
|
set -eu
|
|
command -v zstd >/dev/null 2>&1 || { apt-get update -qq && apt-get install -y -qq zstd; }
|
|
indexexport -o /tmp/core-index.db -artists "$ARTISTS"
|
|
zstd -19 -T0 -q -f /tmp/core-index.db -o /tmp/core-index.db.zst
|
|
sha256sum /tmp/core-index.db.zst | tee /tmp/core-index.db.zst.sha256
|
|
ls -lh /tmp/core-index.db.zst
|
|
|
|
# Nothing is published until it has been imported by the code that
|
|
# imports it on a user's machine. The exporter and the importer are
|
|
# two descriptions of one storage format, and every other tier tests
|
|
# the importer against a *fixture* rather than against the file being
|
|
# shipped — a second description free to be wrong in the same
|
|
# direction as the code reading it.
|
|
#
|
|
# That is how #258 reached everyone: the importer positioned its batch
|
|
# walk with a Go `string` cursor against this file's 16-byte `mbid`
|
|
# column, and SQLite neither coerces between TEXT and BLOB nor
|
|
# complains about the comparison — so the walk merged no rows and
|
|
# never advanced, and no install could finish its first index build.
|
|
# The fixture guarding that walk writes the old text encoding, and the
|
|
# only compact fixture is one row, below the batch size, so the bound
|
|
# query never ran. Both were green throughout.
|
|
#
|
|
# Running it here is also what keeps the failure cheap: the previous
|
|
# artifact stays published while this runs, so a failure costs one
|
|
# stale catalog rather than an empty one for every install.
|
|
#
|
|
# `-tags indexbuild` because this container has no GTK and the default
|
|
# tag set links the app through Wails. The `--- PASS` grep is not
|
|
# decoration — the test skips without the path, and a skip is
|
|
# indistinguishable from a pass in a summary line.
|
|
- name: Import the exported artifact as a client does
|
|
if: steps.maintain.outputs.complete == 'true' && steps.maintain.outputs.changed == 'true'
|
|
working-directory: /src
|
|
env:
|
|
YJ_CORE_INDEX_ARTIFACT: /tmp/core-index.db.zst
|
|
run: |
|
|
set -eu
|
|
log=/tmp/import-check.log
|
|
if ! go test -tags indexbuild -count=1 -timeout 30m -v \
|
|
-run TestImportPublishedArtifact ./backend/explore/ > "$log" 2>&1;
|
|
then
|
|
tail -60 "$log"
|
|
echo "::error::The artifact does not import; not publishing it."
|
|
exit 1
|
|
fi
|
|
cat "$log"
|
|
grep -qF -- 'PASS: TestImportPublishedArtifact' "$log"
|
|
echo "::notice::The artifact imports as a client would merge it."
|
|
|
|
- name: Publish to the Gitea package registry
|
|
if: steps.maintain.outputs.complete == 'true' && steps.maintain.outputs.changed == 'true'
|
|
run: |
|
|
set -eu
|
|
pkg="${SERVER_URL}/api/packages/${OWNER}/generic/yellowjacket-core-index"
|
|
|
|
# Published twice: under a dated version for history, and under
|
|
# the fixed "latest" version the client fetches. Clients cannot
|
|
# discover the newest dated version on their own — the package
|
|
# listing API requires a token, while a plain file GET does not
|
|
# — so "latest" is what makes an anonymous first run possible.
|
|
#
|
|
# A generic package rejects re-uploading a filename that already
|
|
# exists, so "latest" is deleted before being rewritten. It is
|
|
# absent on the very first publish, hence the tolerated 404.
|
|
curl --silent --show-error --user "${OWNER}:${PACKAGE_TOKEN}" \
|
|
--request DELETE "${pkg}/latest" || true
|
|
|
|
for version in "$(date -u +%Y%m%d)" latest; do
|
|
for f in core-index.db.zst core-index.db.zst.sha256; do
|
|
echo "Uploading $f -> $version"
|
|
curl --fail-with-body --user "${OWNER}:${PACKAGE_TOKEN}" \
|
|
--upload-file "/tmp/$f" "${pkg}/${version}/${f}"
|
|
done
|
|
done
|
|
|
|
- name: Summary
|
|
if: always()
|
|
run: |
|
|
echo "complete=${{ steps.maintain.outputs.complete }}"
|
|
echo "changed=${{ steps.maintain.outputs.changed }}"
|
|
if [ "${{ steps.maintain.outputs.complete }}" != "true" ]; then
|
|
echo "Build incomplete — rerun to continue from the checkpoint."
|
|
echo "Progress lives in /cache/data/explore-staging."
|
|
elif [ "${{ steps.maintain.outputs.changed }}" != "true" ]; then
|
|
echo "Nothing new to publish."
|
|
fi
|