yellowjacket/.planning/plans/completed/001-ship-core-index.md

# 001 — Ship a prebuilt "core" explore index

**Status:** complete
**Branch:** cleanup/fresh-start-schema
**Created:** 2026-07-25
**Completed:** 2026-07-30

## Outcome

A fresh install downloads a 70.6 MB artifact and merges 1,076,133 rows
in ~43 s, instead of streaming 205 GB over ~27 h. The dump importer that
produces the artifact left the app binary entirely — it is behind the
`indexbuild` build tag and runs only in CI.

Phase 5 landed differently than planned: rather than a user-facing
setting gating the deep import, the deep import is simply not in the
app. `deep_catalog_enabled` existed briefly and was removed with it.

Two things remain unverified or undone, both recorded in
`.planning/NOTES.md`: anonymous package download on git.ljones.me has
not been confirmed against a real published artifact, and installs whose
index was built by older code (no `listens_applied_series`) have no
rescue path — though with no migration chain, those databases are now
unsupported anyway.

## Problem

A fresh install has no explore index. `StartIndexBuild()` is called
unconditionally from two places in `app.go`, and `runDumpBuild` then
downloads gigabytes from `data.metabrainz.org` before Explore can return
anything beyond the user's own library:

| Stage | Source | Cost |
|---|---|---|
| Listen Counts | ListenBrainz spark full listens dump | **~205 GB streamed** — see below |
| Catalog Import | MusicBrainz canonical dump (~2 GB `.tar.zst`) | scan ~30M CSV rows, assemble to budget |
| Metadata Patch | MB/LB API | rate-limited at 3 req/s |
| Listener Counts | LB API | rate-limited |

Measured 2026-07-25 against the live dump
(`listenbrainz-spark-dump-2593-20260712-000004-full.tar`):

```
content-length: 205073162240   # 205 GB
accept-ranges: bytes
```

The stage-1 reader skips non-`.parquet` tar members
(`dumpcounts.go:317`), but a tar stream has no seek — skipped bytes
still transit the wire. **So a first run on a fresh install pulls
~205 GB.** Little of it touches disk (the counts map and checkpoint do,
not the dump), but the bandwidth is real and it is per-user.

Consequences today:

- Every install pulls ~205 GB to derive a catalog that is **identical
  for everyone**. On a metered or slow connection this is untenable, and
  it is unconditional on first run.
- **It refuses to start without 6 GB free** (`dumpMinStartFreeBytes`),
  and aborts below 2 GB (`dumpAbortFreeBytes`). This is what breaks
  `make fresh-install` on a tmpfs `/tmp`.
- First-run Explore is empty for the length of the import.

The catalog half is **the same for everyone**. Only the local half
(`PopulateLocalCrossReferences`, `BackfillLibraryDiscographies`) is
per-user. Deriving the shared half on each machine is the waste this
plan removes.

## Goal

Ship a prebuilt core index so a fresh install has a usable Explore
immediately, and the runtime build collapses to the local half plus
incremental refresh. The full dump import becomes an opt-in "deep
catalog" upgrade rather than a prerequisite.

## Sizing evidence

Measured 2026-07-25 with a synthetic harness against the real schema and
migrations (2.15M-row full run exceeded a 15-minute budget, so this is a
200K-row calibration, `VACUUM`ed):

| Metric | Value |
|---|---|
| 200,000 rows, with FTS | 85.2 MB |
| Cost per row | ~426 B |
| zstd -19 | 29.6 MB (2.9x) |

Extrapolating to the current budgets (`keepRecordings` 1.5M +
`keepReleaseGroup` 400K + `keepArtists` 250K = 2.15M rows):

| Tier | Rows | On disk | zstd -19 |
|---|---|---|---|
| Full budget | 2.15M | **~900 MB** | ~310 MB |
| Core (proposed) | 500K | ~210 MB | **~72 MB** |
| Minimal | 250K | ~105 MB | ~36 MB |

**This corrects an earlier figure.** A ~93 MB index was recorded in the
2026-07-16 audit note; that measured the *legacy tier-crawl* index, not
the dump-built one. The dump build targets an order of magnitude more
rows. Shipping the full index is not viable as a casual download —
which is exactly why this plan is scoped to a *core* subset.

⚠️ Two caveats on these numbers:

- The harness used a 14-word vocabulary, so its FTS measured only 7% of
  total size. Real titles have a far larger vocabulary and the real FTS
  share will be materially higher. **Treat the totals as a floor.**
- Row width was estimated from the schema (3 UUIDs at 36 chars dominate);
  `aliases` was left empty and is populated for real artists.

Re-measure against a genuine dump-built index before committing to a
tier size.

## What "core" should mean

`dumpcatalog.go` already has graded per-artist coverage (S2) —
`perArtistArtistBudget = 10_000` split into tiers A/B/C with per-tier
track and release-group caps. The core index should reuse that machinery
rather than invent a second notion of importance:

- **Artists:** top ~50K by listen count.
- **Release groups + recordings:** the S2 per-artist slice for those
  artists (tier A/B/C caps as they stand).
- **Excluded:** the global long tail below the per-artist selection.

Anything not covered still works — it just resolves through the existing
lazy paths (`EnsureArtistDiscography`, `AddFromCache`), which is the
behaviour non-covered artists already get today.

## Distribution: download on first run, not `go:embed`

**Both packaging paths build from source** — the Homebrew formula builds
from a release tarball, the Arch `PKGBUILD` clones the tag. So:

- Committing the artifact to git bloats the repo and every source tarball.
- `go:embed` makes a from-source build require the artifact at build
  time, so source builds would have to download it anyway — and
  `build-prod` runs UPX over the binary, which would be pathological
  with a 70 MB+ embedded blob.

So "ship with the app" should mean **fetch a prebuilt artifact on first
run** from a versioned URL. CI already publishes binary packages to the
Gitea package registry (`.gitea/workflows/arch-package.yml`), so there is
an existing place to host it.

Import path: download `.zst` → decompress → `ATTACH` → `INSERT INTO
explore_index SELECT ...` through the **existing** `upsertBatch` conflict
rules, which already do the right thing (non-empty wins, highest
popularity wins, never clobber a good value with an empty one).

## Artifact contents

Ship the global catalog columns only. These are **per-user** and must be
zeroed in the artifact, then recomputed locally by
`PopulateLocalCrossReferences`:

- `in_library`, `is_similar`
- `local_artist_id`, `local_release_group_id`, `local_recording_id`

`discog_fetched` should ship as `1` for artists whose S2 slice is
included, so the backfill doesn't redundantly re-fetch them.

Also decide per-table whether to include: `similar_artist_map`,
`artist_metadata`, `release_to_rg`. `release_to_rg` in particular may
rival the index in size — measure before including.

**Resolved: the artifact ships no FTS.** Rows are inserted into the
client's own `explore_index`, whose `AFTER INSERT` trigger populates
`explore_index_fts` as a side effect — so shipping a search index would
be pure redundant weight. `cmd/indexexport` builds the artifact without
FTS or triggers accordingly.

## Update strategy

- **Popularity drift** — `dumpincremental.go` already implements
  incremental listens-dump refresh (`RefreshListenCounts`, weekly
  cadence). It applies unchanged on top of a shipped baseline, provided
  `listens_applied_series` is stamped in the artifact so deltas resume
  from the right point.
- **Catalog additions** — new releases arrive via the existing lazy
  per-artist fetches. A refreshed artifact per app release is enough;
  no separate cadence needed.
- **Schema changes** — `schema_version` exists on `explore_index` but is
  noted as dead in the audit. Either wire it up or version the artifact
  filename against the migration number, so an old artifact can't be
  imported into a newer schema.

## Build pipeline: build and cache in Gitea CI

The import is unusually well suited to running as a **series of
time-boxed CI jobs against a persistent cache**, because the resumability
already exists:

- Stage 1 streams over a `resumableReader` that reconnects with HTTP
  `Range` requests, and the live dump advertises `accept-ranges: bytes`.
- `counts.bin` checkpoints `Offset` (absolute byte position) and
  `MemberIdx`, and the applier merges results **in member order** so
  "every checkpoint is a contiguous prefix of the stream"
  (`dumpcounts.go`).
- Stage 2's canonical scan is deliberately restartable wholesale — "cheap
  enough to simply restart after an interruption" (`dumpcatalog.go`).

So a job that hits a runner time limit resumes at its exact byte offset
on the next run. **No single multi-hour job is required** — schedule
N bounded runs and let them converge.

What it needs:

1. **A persistent volume for `explore-staging/` + the DB.** `act_runner`
   uses the Docker backend and job containers are ephemeral, so bind-mount
   a host path (or a named Docker volume) and point `YJ_HOME` at it.
   Prefer this over the Actions cache — cache entries are size-capped and
   awkward at GB scale, and this is a self-hosted runner anyway.
2. **A headless entrypoint** — currently the import only runs from the
   app lifecycle (`StartIndexBuild` via `OnDomReady`). This is a real gap,
   but a small one: `NewSearchIndex(db, lb, artistImg, logger)` takes no
   Wails dependency, and the single `runtime.EventsEmit` in
   `searchindex.go` sits inside `emitStatus`, which already early-returns
   when `runtimeCtx == nil`. A `cmd/indexbuild` that opens the DB and
   calls `StartBuild(context.Background())` — never `SetContext` — should
   work. Verify `scheduleChampionRebuild` in the `StartBuild` defer is
   also Wails-free.
3. **Triggers.** `indexbuild` decides its own mode from index state, so
   every trigger runs the same command: push to `main` and a weekly cron
   both land on a cheap refresh (which no-ops when nothing new is
   published), and the 3-month rebuild fires when the command notices the
   import has aged out.

Then export: subset to core, zero the personal columns, stamp
`dump_import_done` / `listens_applied_series` / schema version, `VACUUM`,
`zstd -19`, checksum, publish to the Gitea package registry (the Arch
workflow already authenticates against it with `PACKAGE_TOKEN`).

**Be a good citizen about the 205 GB.** Rebuild on the dump cadence
(the audit notes a 90-day re-import cadence), never per-commit. Once a
baseline exists, the ~180 MB daily incremental dumps already wired in
`dumpincremental.go` keep popularity fresh — so the 205 GB is genuinely
one-time per rebuild, not per refresh. Also check the runner's own
egress if it is self-hosted on a home connection.

## Licensing

- MusicBrainz canonical dump is **CC0** — redistribution fine.
- ListenBrainz-derived listen counts need their dump licence checked
  before redistribution, plus attribution in-app either way.
- Note the derived counts already differ from LB API values (no MLHD+
  history) — a known, accepted divergence, but worth stating wherever
  the numbers are surfaced.

## Risks

- **Artifact staleness vs app version** — a user on an old release gets
  an old catalog. Mitigated by incremental refresh + lazy fetches.
- **Download failure / offline install** — must degrade to today's
  behaviour (local library search), not a broken Explore. The failure is
  now visible in the Jobs panel, which helps.
- **Users who want the full catalog** — keep the existing dump import as
  an explicit opt-in, gated behind a setting. Note that no such setting
  exists today: `StartIndexBuild()` is unconditional, and Library Only
  mode is frontend-`localStorage` only with no backend wiring.

## Phasing

1. ✅ **Headless entrypoint.** `cmd/indexbuild` — resumable, budgeted
   (`-budget 3h`), signal-aware, exit 3 = "more work remains". Verified
   to run without Wails; builds with `CGO_ENABLED=0` and no build tags.
2. ✅ **Export tooling.** `cmd/indexexport` — top-N artists plus a
   per-artist window of their release groups and recordings, personal
   columns dropped, metadata stamped, vacuumed. Verified against a
   synthetic index: no personal columns leak, no orphaned rows, caps
   respected.
3. ✅ **One real build.** Superseded by a real dump-built index that
   already existed on the dev machine (`dump_import_done` 2026-07-17).
   Measured 2026-07-29 — these replace every extrapolation above:

   | | rows | on disk |
   |---|---|---|
   | `explore_index` | 2,052,168 (227,359 artists / 400,675 RGs / 1,424,134 recordings) | 383 MB |
   | its indexes | | 395 MB |
   | FTS | | 80 MB |

   187 B/row for the shippable table, 418 B/row all-in — so the ~900 MB
   full-budget estimate was right. Two real exports:

   | tier | rows | artifact | zstd -19 |
   |---|---|---|---|
   | 50K artists (default) | 1,076,133 | 191.5 MB | **70.6 MB** |
   | 25K artists / 10 RG / 20 rec | 620,973 | 110.6 MB | **37.8 MB** |

   `release_to_rg` was empty in that index — it predates the code that
   persists it — so its size is still unmeasured.
4. ✅ **Import path.** `backend/explore/artifactfetch.go` (download,
   Range-resume, sha256, zstd) and `artifactimport.go` (validate, ATTACH,
   batched merge, FTS rebuild, meta stamping). Reported in the Jobs panel
   under its own two stages. Measured end to end on the real 50K-artist
   artifact against a disk-backed DB: **1,076,133 rows merged in 43.2s**
   (24,900 rows/s), yielding a 455 MB `yj.db`, FTS populated and
   searchable. In-memory the same merge runs in 28.3s.
5. ✅ **Gate the dump build.** `deep_catalog_enabled` in
   `explore_index_meta` (beside `index_build_paused` — it is build state,
   read at one decision point). Off by default; exposed as
   `DeepCatalogEnabled` / `SetDeepCatalogEnabled` on the explore Service.
   An interrupted dump import resumes regardless of the setting, so the
   gate never discards a checkpoint that already cost hours.

## Measured 2026-07-29: why the client cannot fix this itself

`data.metabrainz.org` caps a client at ~2.1 MB/s. One Range stream and
four concurrent Range lanes both delivered 32 MB at the same aggregate
rate (2,111,195 B/s vs 2,209,000 B/s) while the same machine pulled
66.9 MB/s from a CDN. **Parallelism buys nothing** — the four lanes just
divide the same cap, and one of them starved to 0.5 MB/s.

So stage 1 costs, unavoidably:

| | bytes | wall clock |
|---|---|---|
| Whole tar (what shipped before column projection) | 205 GB | ~27 h |
| Column projection, 3 columns (43.4%) | 89 GB | ~11.8 h |
| `recording_mbid` only (24.1%), rolled up via canonical | 49 GB | ~6.5 h |
| + 1-in-4 member stride sample | 12 GB | ~1.6 h |

The last two are CI-side options, not client defaults: recording-only
drops listens carrying no recording MBID and re-derives artist totals as
a sum over recordings, and sampling trades exact counts for a ranking.
Both are only safe because the selection they feed is a top-N cut.

## Distribution: the "latest" version trick

The client cannot enumerate package versions — Gitea's package listing
API requires a token, while an anonymous file GET does not (a probe of a
non-existent artifact returns 404, not 401). So `index-artifact.yml`
publishes each artifact twice: under a dated version for history, and
under a fixed `latest` version that the client fetches from a
predictable URL. Generic packages reject overwriting an existing
filename, so `latest` is DELETEd before each rewrite.

⚠️ **Unverified:** that anonymous package *download* is actually enabled
on git.ljones.me. The 404-vs-401 probe is suggestive, not proof — no
artifact has been published yet to test against. Confirm before relying
on it, and note that every install pulling from a personal Gitea makes
its bandwidth and uptime a user-facing dependency.

## Incremental retention bounds artifact staleness

The incremental dump directory holds 30 dumps (series 2579–2610 as of
2026-07-29) and full dumps land roughly monthly. An artifact older than
~30 days therefore cannot be topped up: the dailies bridging the gap are
gone. That is a permanent undercount of that window's listens, not
corruption — but it pins the republish cadence at monthly.

## Upgrade path for indexes built by older code

The dev machine's index has `dump_import_done` set but **no**
`listens_applied_series` and an empty `release_to_rg`, because it was
built before the code that writes them. That combination is a dead end:
`RefreshListenCounts` bails with "no baseline series recorded", and
`runDumpBuild` short-circuits on the done marker, so popularity can
never update again. Current code writes both, so this affects only
pre-existing installs — but the artifact import is the natural place to
rescue them, since merging one stamps a fresh baseline series.
6. ✅ **CI wiring.** `.gitea/workflows/index-artifact.yml` — push +
   weekly cron + manual, concurrency-guarded, publishes only when
   `complete && changed` so identical artifacts don't accumulate.
   Runner-side prerequisites are in place (cache dir + `valid_volumes`
   on the VPS runner).

Step 3 is the gate on everything downstream — and it is worth doing
regardless of whether the artifact ever ships, since it is the only way
to get real numbers for the index.

## Related

- `backend/explore/dumpimport.go` — stage orchestration, disk floors
- `backend/explore/dumpcatalog.go` — budgets, S2 per-artist tiers
- `backend/explore/dumpincremental.go` — incremental refresh (update path)
- `backend/explore/searchindex.go` — `upsertBatch` conflict rules,
  `PopulateLocalCrossReferences`
- Migration 26 in `backend/database/database.go` — `explore_index` schema