yonluandClaude Opus 5
4f8257ef72
The stale-shape repair dropped the CI catalog on its first run:
retiring a table ... table=explore_index
reason="column entity_type is TEXT, schema declares INTEGER"
index maintenance mode=build reason="no completed import yet"
The mismatch was real and the drop was correct by the app's rule: a
client's catalog is *downloaded*, so a wrong shape costs a minute of
re-fetching the artifact, while keeping it costs every Explore read.
It is the wrong rule for one database. cmd/indexbuild's catalog is not
downloaded, it is what the artifact is cut from — the only way back is
the ~205 GB dump stream the /cache volume exists to avoid. And that
database is deliberately kept in the older encoding, which
`fix(indexexport): read an index older than the binary` exists to
tolerate, so the shape does not match by design and would have been
dropped on every run.
retireLibraryTables, right beside it, never touches the catalog for
exactly this reason. The repair reached past that protection because it
runs inside database.NewDB, which cmd/indexbuild also calls.
So the policy is a build tag, which is how this project already tells
the index tools apart (runtime_indexbuild.go, servicestartup.go,
dumpbuild_stub.go): Cache tables are rebuilt in the app and never in
cmd/indexbuild. Owned and Derived are still repaired in both — that is
the half this database can safely discard, and retireLibraryTables
already discards it.
The residual trade is deliberate: a future explore_index column will
now fail the index job loudly on applySchema rather than silently
costing it a 205 GB rebuild. A human should decide that one.
TestTheCatalogSurvivesAStaleShape is the accident, symptom first, with
the shape the real database is in — every current column, ids and
entity type still text. It fails with "the catalog was retired" when
the policy is flipped back.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AfVYUVExXsx1nSWrXN8mAh
38 lines
1.7 KiB
Go
38 lines
1.7 KiB
Go
//go:build indexbuild
|
|
|
|
package database
|
|
|
|
// retireStaleCache is false here, and this is the whole reason the
|
|
// policy is a build tag rather than a rule inside retireStaleTables.
|
|
//
|
|
// The index database is the one place in this project where the catalog
|
|
// is *derived* rather than downloaded. Rebuilding it is a ~205 GB dump
|
|
// stream over hours, resumed across runs from a checkpoint on a
|
|
// persistent volume; that volume exists for no other purpose. The app's
|
|
// answer to a stale catalog — drop it, fetch the artifact again — is
|
|
// not available here, because this database *is* what the artifact is
|
|
// cut from.
|
|
//
|
|
// This was not hypothetical. The repair shipped without it and dropped
|
|
// the CI catalog on its first run:
|
|
//
|
|
// retiring a table ... table=explore_index
|
|
// reason="column entity_type is TEXT, schema declares INTEGER"
|
|
// index maintenance mode=build reason="no completed import yet"
|
|
//
|
|
// The shape mismatch was real and the drop was correct by the app's
|
|
// rule. It was still wrong here: that database is deliberately kept in
|
|
// the older encoding, which is what `fix(indexexport): read an index
|
|
// older than the binary` exists to tolerate. A rule that is right for
|
|
// every install and catastrophic for one database has to be told which
|
|
// one it is in, and a build tag is how this project already tells the
|
|
// index tools apart (backend/events/runtime_indexbuild.go,
|
|
// backend/explore/servicestartup.go, dumpbuild_stub.go).
|
|
//
|
|
// cmd/indexbuild has its own repair for the half it *can* safely
|
|
// discard: retireLibraryTables drops every table the datamap does not
|
|
// classify as Cache, which is empty by construction in that database.
|
|
// Between the two, the library half is repaired and the catalog is
|
|
// never touched.
|
|
const retireStaleCache = false
|