ci: trigger the catalog job deliberately, pin agent docs to one file
CI / check (pull_request) Canceled after 0s
CI / e2e (pull_request) Canceled after 0s
CI / check (push) Canceled after 0s
CI / e2e (push) Canceled after 0s

Two guardrails for the 2026-08-17 incident, and one is not about CI.

index-artifact.yml's `push` trigger was commented out that day with a
note to restore it once the rebuild completed. Restoring it is the bug.
A refresh is individually cheap, which is what made the trigger look
free; what it actually did was put an unattended job that mutates the
only copy of a ~205 GB catalog on the same trigger as an ordinary code
change, on a runner with capacity 1. The rule the file now states is the
general one -- a job that mutates state which cannot be rebuilt in ten
minutes is triggered deliberately -- so the next such job has somewhere
to look. The cron and workflow_dispatch lose nothing: indexbuild resumes
from its checkpoint either way.

Note what no branching or PR gate would have caught here. That change
was green on its branch, green on the merge and green on main; the fault
existed only against the persistent /cache database, which no fixture
reproduces. Code is gated by CI, irreplaceable state by refusing to
touch it and by docs/index-cache.md's restore.

The other half is the mismatch that started this: two harnesses reading
two files. AGENTS.md is a symlink to CLAUDE.md and skill-check asserts
the symlink rather than comparing contents, because a copy would satisfy
every other check in this repo while silently drifting -- which is the
failure being prevented. The same check now scans CLAUDE.md for make
targets, which it never did: 27 targets named in the file agents trust
most, none of them verified. Coverage goes 19 -> 46.

Scanning prose meant the line-start rule needed a fence. "Two green
branches do not / make a green merge" wrapped onto a line beginning
`make a` and duly failed on a target called `a`. Inside a fence it is
code; outside one it is a sentence that broke there, and a check that
fails on reflow gets disabled rather than fixed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AfVYUVExXsx1nSWrXN8mAh
This commit is contained in:
2026-08-17 15:30:38 -04:00
co-authored by Claude Opus 5
parent 3d65da0529
commit e51cb13662
4 changed files with 98 additions and 40 deletions
+20 -22
View File
@@ -7,31 +7,29 @@ name: Search index maintenance
# import older than 6mo -> rebuild (re-import from the newest dump)
# otherwise -> refresh (fold in new incremental listens)
#
# A refresh is cheap and no-ops when nothing new has been published, so
# running it on every push to main is safe.
# **There is deliberately no `push` trigger, and restoring one is a
# decision rather than a cleanup.** A refresh is individually cheap, so
# running it on every push to main looked free; what it actually does is
# put an unattended job that mutates the only copy of a ~205 GB catalog
# on the same trigger as an ordinary code change, on a runner with
# capacity 1.
#
# ---------------------------------------------------------------------
# TEMPORARY (2026-08-17): the `push` trigger is off while the catalog
# rebuilds.
# That is not hypothetical. On 2026-08-17 `fix(database): retire a table
# whose shape the schema moved past` landed on main, green — the CI
# database is deliberately in the older encoding, so the stale-shape
# repair judged its `explore_index` stale and dropped it, and this job
# fell back to a full import from the dumps. `fix(database): never
# retire the catalog the index build derives` stops that specific repair
# and cannot undo it. Every push to main then booked another `budget`
# (3h) of the one runner while ordinary CI queued behind it.
#
# `fix(database): retire a table whose shape the schema moved past`
# dropped this job's `explore_index` on its first run -- the CI database
# is deliberately in the older encoding, so the repair judged it stale --
# and the job fell back to a full ~205 GB import from the dumps.
# `fix(database): never retire the catalog the index build derives`
# stops it happening again but cannot undo it.
#
# Until that import reports complete, every push to main books another
# `budget` (3h) of a runner with capacity 1, and ordinary CI queues
# behind it. The weekly cron and workflow_dispatch still resume the
# build, which is all it needs: indexbuild picks up from its checkpoint.
#
# RESTORE the two `push` lines below once a run reports
# `complete=true`. Nothing else here changed.
# ---------------------------------------------------------------------
# So the rule this file is an instance of: **a job that mutates state
# which cannot be rebuilt in ten minutes is triggered deliberately, not
# by a push.** The weekly cron keeps the catalog current, and
# workflow_dispatch resumes or forces a build — indexbuild picks up from
# its checkpoint either way, so nothing is lost by not running on every
# merge. See docs/index-cache.md for the snapshot and the restore.
on:
# push:
# branches: [main]
schedule:
# Weekly update pass. The 6-month rebuild is triggered by the same
# command when it notices the import has aged out.
Symlink
+1
View File
@@ -0,0 +1 @@
CLAUDE.md
+6 -4
View File
@@ -179,10 +179,12 @@ bindings-check: ## Fail if the generated bindings are stale
css-check: ## Fail if a css`` literal was ended early by a backtick in a comment
@cd frontend && node scripts/check-css-literals.mjs
# .pi/ documents commands, and a skill that documents a command wrongly
# is worse than no skill: an agent runs it confidently. Every command
# in there is a make target on purpose, so this is checkable.
skill-check: ## Fail if .pi/ documents a make target that does not exist
# .pi/ and CLAUDE.md document commands, and a doc that documents a
# command wrongly is worse than no doc: an agent runs it confidently.
# Every command in them is a make target on purpose, so this is
# checkable. It also asserts AGENTS.md is a symlink to CLAUDE.md, so the
# two harnesses cannot drift onto two descriptions of one project.
skill-check: ## Fail if the agent docs name a missing make target, or AGENTS.md is not a symlink
@./scripts/skill-check.sh
# Conventional Commits, which CLAUDE.md claimed CI enforced for a long
+71 -14
View File
@@ -1,17 +1,48 @@
#!/usr/bin/env bash
#
# Every command in .pi/ is a `make` target on purpose: the Makefile is
# the source of truth for *how* to invoke something, and the skill only
# decides *which* and *in what order*. This check keeps that honest —
# a renamed or deleted target turns into a failing commit rather than
# into an agent confidently running a command that no longer exists.
# Every command in the agent-facing docs is a `make` target on purpose:
# the Makefile is the source of truth for *how* to invoke something, and
# the docs only decide *which* and *in what order*. This check keeps
# that honest — a renamed or deleted target turns into a failing commit
# rather than into an agent confidently running a command that no longer
# exists.
#
# It extracts every `make <target>` mentioned under .pi/ and asserts the
# target exists. Usage: scripts/skill-check.sh
# It checks two things. Usage: scripts/skill-check.sh
#
# **Every `make <target>` named in an agent-facing doc exists.** The
# scanned set is `.pi/` *and* CLAUDE.md, which is the half that was
# missing: CLAUDE.md names 27 targets and nothing verified one of them,
# so the file the agents trust most was the file least checked.
#
# **AGENTS.md is a symlink to CLAUDE.md.** This repo is worked on by
# two agent harnesses that read different files by convention — Claude
# Code reads CLAUDE.md, others read AGENTS.md — and two harnesses
# reading two descriptions of one project is how they come to hold
# different beliefs about it. A symlink makes that impossible by
# construction; a *copy* would pass every other check in this repo while
# silently drifting, which is exactly the failure being prevented, so
# the symlink itself is asserted rather than its contents compared.
set -euo pipefail
cd "$(dirname "$0")/.."
# The symlink half runs even without .pi/, since it is not about .pi/.
if [ -e AGENTS.md ] || [ -L AGENTS.md ]; then
if [ ! -L AGENTS.md ]; then
echo "skill-check: AGENTS.md is a regular file, not a symlink to CLAUDE.md." >&2
echo " Two harnesses would read two descriptions of one project." >&2
echo " Fix: rm AGENTS.md && ln -s CLAUDE.md AGENTS.md" >&2
exit 1
fi
target="$(readlink AGENTS.md)"
if [ "$target" != "CLAUDE.md" ]; then
echo "skill-check: AGENTS.md points at '$target', expected CLAUDE.md." >&2
exit 1
fi
fi
[ -d .pi ] || exit 0
# `make -pq` prints the database including every rule, without running
@@ -21,11 +52,37 @@ targets="$({ make -pqRr 2>/dev/null || true; } |
awk '/^[a-zA-Z0-9][^$#\/\t=]*:([^=]|$)/ {sub(/:.*/, "", $0); print}' |
sort -u)"
# A mention counts only when it is code: backticked (`make ui-test`) or
# the first thing on a line, as in a fenced block. Bare prose is not
# scanned, because English says things like "a renamed make target".
mentioned="$(grep -rhoE '(`|^)make [a-z][a-z0-9-]*' .pi --include='*.md' |
sed 's/^`//' | awk '{print $2}' | sort -u)"
# A mention counts only when it is code: backticked (`make ui-test`)
# anywhere, or at the start of a line **inside a fenced block**. Bare
# prose is not scanned, because English says things like "a renamed make
# target".
#
# The fence is why this is awk rather than one grep. Line-start alone is
# not evidence of code in a file that is mostly hard-wrapped prose: the
# sentence "Two green branches do not / make a green merge" wrapped onto
# a line beginning `make a`, and the check duly failed on a target called
# `a`. Inside a fence it is code; outside one it is a sentence that
# happened to break there, and a check that fails on reflow gets
# disabled rather than fixed.
#
# AGENTS.md is deliberately not in this list: it is a symlink to
# CLAUDE.md, asserted above, so scanning it would report every failure
# twice under two names.
mentioned="$({ find .pi -name '*.md' 2>/dev/null; echo CLAUDE.md; } |
xargs awk '
FNR == 1 { fence = 0 }
/^```/ { fence = !fence; next }
{
rest = $0
while (match(rest, /`make [a-z][a-z0-9-]*/)) {
print substr(rest, RSTART + 6, RLENGTH - 6)
rest = substr(rest, RSTART + RLENGTH)
}
if (fence && match($0, /^make [a-z][a-z0-9-]*/)) {
print substr($0, 6, RLENGTH - 5)
}
}
' | sort -u)"
missing=""
@@ -36,10 +93,10 @@ for t in $mentioned; do
done
if [ -n "$missing" ]; then
echo "skill-check: .pi/ documents make targets that do not exist:" >&2
echo "skill-check: the agent docs name make targets that do not exist:" >&2
for t in $missing; do
echo " make $t" >&2
grep -rln "make $t" .pi --include='*.md' | sed 's/^/ /' >&2
grep -rln "make $t" .pi CLAUDE.md --include='*.md' | sed 's/^/ /' >&2
done
echo "Fix the docs, or restore the target." >&2
exit 1