yonlu/yellowjacket
1
1
Fork 0
Code Issues 53 Pull Requests 1 Actions Packages Projects Releases 13 Wiki Activity

Explore: the prebuilt catalog never merges into the index — the batch walk compares a TEXT cursor against the artifact's BLOB mbid column and loops forever #258

New Issue
Open
opened 2026-09-25 13:47:56 +00:00 by yonlu · 1 comment
yonlu commented 2026-09-25 13:47:56 +00:00
Owner
Copy Link
Copy Source

Symptom

On a fresh sandbox (make sandbox <name>), the "Building search index" job runs
forever: the progress bar sits at 0 of 1,077,893 rows on the "Merging
prebuilt catalog" stage, the app burns a core indefinitely (48 h of CPU across
~12 threads in the reproduction), and no catalog row is ever merged —
explore_index holds only the rows the library scan and the lazy artist
enrichment created. Explore therefore finds nothing catalog-only, and
popularity data is absent for everything the artifact would have covered.

The library scan is unaffected, which is what makes it look like a cosmetic
progress bug rather than a broken import.

Cause

mergeArtifactRows walks the artifact in batches, positioning itself with:

SELECT mbid FROM core.explore_index
 WHERE mbid > ? ORDER BY mbid LIMIT 1 OFFSET ?

with the cursor passed as a Go string, while cmd/indexexport publishes
explore_index.mbid as BLOB (16 raw bytes — the storage change that took
the table from 677 MB to 389 MB).

SQLite does not coerce between TEXT and BLOB, and orders every blob after every
text value, so against a byte column:

  • mbid > '' matches the whole table, so ORDER BY mbid LIMIT 1 OFFSET 49999 returns the same row every time and the cursor never advances;
  • mbid <= <text> matches no rows, so every batch merges 0 rows.

The loop is therefore unbounded and silent: no error, no rows, no state change,
and a progress bar that never moves. The same defect in the other direction
(blob cursor against the older text artifact) is equally unbounded.

It was never caught because the fixture that guards the walk
(writeTestArtifact) writes the text encoding, and the only compact fixture
(TestImportCoreArtifactAcceptsBothEncodings) is a single row — below
artifactMergeBatch, so the bound query never runs.

Fix

  • Bind the cursor in the artifact's own encoding, decided by the existing
    artifactStoresText() probe rather than by a version number.
  • Fail loudly if the walk's bound does not strictly advance, so a future
    encoding mismatch is a failed job with a reason instead of an infinite spin.

Verification

  • TestImportCoreArtifactBatchWalkCoversAllRowsCompact — a compact artifact
    crossing several batch boundaries. Times out against the pre-fix build.
  • Merge of the real published artifact (1,077,893 rows) into a scratch index.
## Symptom On a fresh sandbox (`make sandbox <name>`), the "Building search index" job runs forever: the progress bar sits at `0 of 1,077,893 rows` on the "Merging prebuilt catalog" stage, the app burns a core indefinitely (48 h of CPU across ~12 threads in the reproduction), and **no catalog row is ever merged** — `explore_index` holds only the rows the library scan and the lazy artist enrichment created. Explore therefore finds nothing catalog-only, and popularity data is absent for everything the artifact would have covered. The library scan is unaffected, which is what makes it look like a cosmetic progress bug rather than a broken import. ## Cause `mergeArtifactRows` walks the artifact in batches, positioning itself with: ```sql SELECT mbid FROM core.explore_index WHERE mbid > ? ORDER BY mbid LIMIT 1 OFFSET ? ``` with the cursor passed as a **Go `string`**, while `cmd/indexexport` publishes `explore_index.mbid` as **`BLOB`** (16 raw bytes — the storage change that took the table from 677 MB to 389 MB). SQLite does not coerce between TEXT and BLOB, and orders every blob after every text value, so against a byte column: * `mbid > ''` matches the **whole table**, so `ORDER BY mbid LIMIT 1 OFFSET 49999` returns the same row every time and the cursor never advances; * `mbid <= <text>` matches **no rows**, so every batch merges 0 rows. The loop is therefore unbounded and silent: no error, no rows, no state change, and a progress bar that never moves. The same defect in the other direction (blob cursor against the older text artifact) is equally unbounded. It was never caught because the fixture that guards the walk (`writeTestArtifact`) writes the **text** encoding, and the only compact fixture (`TestImportCoreArtifactAcceptsBothEncodings`) is a **single row** — below `artifactMergeBatch`, so the bound query never runs. ## Fix - Bind the cursor in the artifact's own encoding, decided by the existing `artifactStoresText()` probe rather than by a version number. - Fail loudly if the walk's bound does not strictly advance, so a future encoding mismatch is a failed job with a reason instead of an infinite spin. ## Verification - `TestImportCoreArtifactBatchWalkCoversAllRowsCompact` — a compact artifact crossing several batch boundaries. Times out against the pre-fix build. - Merge of the real published artifact (1,077,893 rows) into a scratch index.
yonlu self-assigned this 2026-09-25 13:48:03 +00:00
yonlu added the
Status
In Progress
 label 2026-09-25 13:48:04 +00:00
yonlu commented 2026-09-25 13:48:04 +00:00
Author
Owner
Copy Link
Copy Source

Taking this. Branch fix/258-artifact-blob-cursor.

Approach: type the walk cursor to the artifact's own encoding (already probed
by artifactStoresText), add a strictly-advancing guard so a mismatch fails the
job instead of spinning, and cover the batch walk on a compact artifact — the
fixture that guarded it writes the old text encoding, and the compact one is a
single row, below the batch size, so the bound query never ran.

Taking this. Branch `fix/258-artifact-blob-cursor`. Approach: type the walk cursor to the artifact's own encoding (already probed by `artifactStoresText`), add a strictly-advancing guard so a mismatch fails the job instead of spinning, and cover the batch walk on a compact artifact — the fixture that guarded it writes the old text encoding, and the compact one is a single row, below the batch size, so the bound query never ran.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Area/Design
Area/Downloads
Area/Explore
Area/Library-UI
Area/Metadata
Area/Packaging
Area/Player
Area/Queue
Area/Settings
Area/Shell-Nav
Compat/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
 Platform/Android
Platform/Desktop
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Medium
3
The priority is medium
Priority
Low
4
The priority is low
Reviewed
Confirmed
1
Issue has been confirmed
Reviewed
Duplicate
2
This issue or pull request already exists
Reviewed
Invalid
3
Invalid issue
Reviewed
Won't Fix
3
This issue won't be fixed
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status
In Progress
Somebody is actively working on this right now
No labels Area/Explore Kind/Bug Platform/Android Platform/Desktop
Priority
Critical
1
Reviewed
Confirmed
1
Status
In Progress
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
logan yonlu
No Assignees yonlu
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: yonlu/yellowjacket#258