yonlu/yellowjacket
1
1
Fork 0
Code Issues 53 Pull Requests 1 Actions Packages Projects Releases 13 Wiki Activity

Explore: the catalog baseline is imported once and never updated — nothing compares the published version to the one on disk #261

New Issue
Open
opened 2026-09-25 15:10:56 +00:00 by yonlu · 0 comments
yonlu commented 2026-09-25 15:10:56 +00:00
Owner
Copy Link
Copy Source

What happens

The catalog artifact is imported once, ever. runDumpBuild (the app's
dumpbuild_stub.go) short-circuits on two bare markers:

if si.hasMeta(dumpImportDoneKey) { ... return }   // stamped by the merge
if si.artifactAlreadyMerged() { return }          // bare hasMeta(core_artifact_version)

Nothing compares the stored core_artifact_version (or the artifact's
built_at) against the published one, so an install that has merged an
artifact never looks at another one. PrepareIndexRebuild — a manual action —
is the only route back.

The published artifact is on a weekly cadence, and the registry currently holds
20260824, 20260831, 20260914, 20260921 and latest. So every one of
those — new artists above indexexport's cut, corrected metadata, a changed
--artists coverage — reaches new installs only.

Why it matters more than it looks

dumpincremental.go keeps popularity current, and that works, so the index
does not look frozen. What never updates is coverage and catalog metadata, which
is the half the artifact exists for: an artist that entered the top 50,000 in
September is unfindable in Explore on every install older than that, with no
signal that anything is stale.

It also makes #258's sibling observation load-bearing: bundling the artifact
with the app would buy an existing install nothing today, because both
short-circuits fire before anything opens the bundled file. Whichever way the
baseline travels, the version comparison has to exist first.

Options

  1. Compare and re-import. The merge is already idempotent by construction —
    upsert by MBID, non-empty wins, higher popularity wins — so importing a newer
    baseline over an older one needs no teardown, and it measured ~50 s for
    1,077,893 rows     on a real library. The question is policy, not mechanism:
    re-import on every launch (a registry HEAD each time), on a cadence, when the
    user asks, or when the app updates (which is what bundling would give).
  2. Expose it. "Catalog built 2026-08-24" with a Refresh button in Settings →
    Search Index, which is honest and costs no automatic traffic.
  3. Both: the deepest correct behaviour and a visible state to check it in.

Not to do

The blank skip must stay correct: hasMeta(dumpImportDoneKey) is also what stops
a full dump import being triggered on a client that already has the catalog, and
the indexbuild build shares that function. A version comparison belongs beside
the skip, not in place of it.

## What happens The catalog artifact is imported **once, ever**. `runDumpBuild` (the app's `dumpbuild_stub.go`) short-circuits on two bare markers: ```go if si.hasMeta(dumpImportDoneKey) { ... return } // stamped by the merge if si.artifactAlreadyMerged() { return } // bare hasMeta(core_artifact_version) ``` Nothing compares the stored `core_artifact_version` (or the artifact's `built_at`) against the published one, so an install that has merged an artifact never looks at another one. `PrepareIndexRebuild` — a manual action — is the only route back. The published artifact is on a weekly cadence, and the registry currently holds `20260824`, `20260831`, `20260914`, `20260921` and `latest`. So every one of those — new artists above `indexexport`'s cut, corrected metadata, a changed `--artists` coverage — reaches new installs only. ## Why it matters more than it looks `dumpincremental.go` keeps **popularity** current, and that works, so the index does not look frozen. What never updates is coverage and catalog metadata, which is the half the artifact exists for: an artist that entered the top 50,000 in September is unfindable in Explore on every install older than that, with no signal that anything is stale. It also makes #258's sibling observation load-bearing: **bundling the artifact with the app would buy an existing install nothing today**, because both short-circuits fire before anything opens the bundled file. Whichever way the baseline travels, the version comparison has to exist first. ## Options 1. Compare and re-import. The merge is already idempotent by construction — upsert by MBID, non-empty wins, higher popularity wins — so importing a newer baseline over an older one needs no teardown, and it measured **~50 s for 1,077,893 rows** on a real library. The question is policy, not mechanism: re-import on every launch (a registry HEAD each time), on a cadence, when the user asks, or when the *app* updates (which is what bundling would give). 2. Expose it. "Catalog built 2026-08-24" with a Refresh button in Settings → Search Index, which is honest and costs no automatic traffic. 3. Both: the deepest correct behaviour *and* a visible state to check it in. ## Not to do The blank skip must stay correct: `hasMeta(dumpImportDoneKey)` is also what stops a full dump import being triggered on a client that already has the catalog, and the `indexbuild` build shares that function. A version comparison belongs beside the skip, not in place of it.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Area/Design
Area/Downloads
Area/Explore
Area/Library-UI
Area/Metadata
Area/Packaging
Area/Player
Area/Queue
Area/Settings
Area/Shell-Nav
Compat/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
 Platform/Android
Platform/Desktop
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Medium
3
The priority is medium
Priority
Low
4
The priority is low
Reviewed
Confirmed
1
Issue has been confirmed
Reviewed
Duplicate
2
This issue or pull request already exists
Reviewed
Invalid
3
Invalid issue
Reviewed
Won't Fix
3
This issue won't be fixed
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status
In Progress
Somebody is actively working on this right now
No labels Area/Explore Kind/Bug Platform/Android Platform/Desktop
Priority
Medium
3
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
logan yonlu
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: yonlu/yellowjacket#261