ci: stop booking three hours of runner on every push
Build & publish Arch package / arch-package (push) Successful in 2m34s
CI / check (push) Canceled after 53s
CI / e2e (push) Canceled after 0s

The catalog this job derives was dropped by the stale-shape repair (see
`fix(database): never retire the catalog the index build derives`, which
prevents a recurrence but cannot undo it), so `mode=auto` now resolves to
a full ~205 GB import from the dumps.

That import runs on every push to main with a 3h budget, on a runner of
capacity 1 -- so ordinary CI has been queuing behind it since the merge,
and each further push books another three hours. The damage is the
repetition, not the single job.

The `push` trigger is commented out until a run reports `complete=true`.
The weekly cron and workflow_dispatch still resume the build, which is
all it needs: indexbuild picks up from its checkpoint, so nothing already
imported is re-fetched.

Restoring the two commented lines is the entire revert, and the comment
beside them says so. NOTES.md carries the incident, including the two
things worth changing regardless: a destructive repair running inside
`database.NewDB` has to ask which binary it is in, and the only copy of a
205 GB derived asset is a single Docker volume with no snapshot.
This commit is contained in:
2026-08-17 13:29:58 -04:00
parent 4f8257ef72
commit 1c4d6ca9a1
2 changed files with 88 additions and 2 deletions
+66
View File
@@ -3338,3 +3338,69 @@ Four things measured or corrected in the doing:
- **The gate belongs before the first status write.** Declining is a
no-op — no job in the indicator, no error tier to dismiss — which is
what makes the refusal safe to have on by default.
## The stale-shape repair dropped the CI catalog (2026-08-17)
Not our change, but it is the operational state everything else now runs
in, and the restore condition needs to be written down somewhere that is
not a commit message.
`fix(database): retire a table whose shape the schema moved past` added
`staleshape.go`: before `applySchema`, drop any non-Authored table whose
live shape disagrees with the schema. That is the right rule for an
install — a client's catalog is *downloaded*, so a stale one costs a
minute of re-fetching the artifact, and keeping it costs every Explore
read.
It runs inside `database.NewDB`, which `cmd/indexbuild` also calls. On
the first run after it landed, 19 seconds in:
```
16:15:51 retiring a table ... table=explore_index
reason="column entity_type is TEXT, schema declares INTEGER"
16:16:05 index maintenance mode=build reason="no completed import yet"
lastImported=never baselineSeries=0
```
**The premise was false for the one database where it was expensive.**
That catalog is not stale; it is deliberately kept in the older text
encoding, which `artifactStoresText` and `sourceColumns` exist to
tolerate — so it would have been judged stale and dropped on *every*
run. And `retireLibraryTables`, in the same package, already documents
the opposite rule for this database: drop everything the datamap does
**not** call Cache.
`fix(database): never retire the catalog the index build derives` makes
the policy a build tag (`retireStaleCache`, false under `indexbuild`),
which is how this project already separates the index tools. It prevents
recurrence and cannot undo the drop: that volume was the only copy.
**What it cost, and the shape of the cost.** A full re-import from the
MetaBrainz dumps, resumed across runs from a checkpoint, at a rate that
swung between 2 and 15 MB/s. The job runs on **every push to main** with
a 3 h budget on a runner of capacity 1 — so until the import completes,
every push books three hours and ordinary CI queues behind it. That is
the real damage: not one lost job, but a repeating one.
So the `push:` trigger in `index-artifact.yml` is **commented out**
until a run reports `complete=true`; the weekly cron and
`workflow_dispatch` still resume the build, which is all it needs.
Restoring those two lines is the whole revert.
Three things worth keeping from it:
- **A repair belongs where its assumptions hold.** `NewDB` is the one
chokepoint every binary in this project shares, including the one
whose database cannot be re-derived cheaply. Anything destructive
there needs to ask which binary it is in — the build tag was available
and is what the fix used.
- **The only copy of a 205 GB derived asset is one Docker volume.**
There is no snapshot, so the restore time is "however long
MetaBrainz takes today". A periodic copy would turn this class of
incident into twenty minutes.
- **The fix's residual trade is now the thing to watch**: with Cache
tables never retired under `indexbuild`, a future `explore_index`
column fails that job loudly at build time instead of silently
rebuilding. That is the right default, and it means the next schema
change touching `explore_index` needs a deliberate plan for this one
database rather than none.