yonlu/yellowjacket
1
1
Fork 0
Code Issues 55 Pull Requests 6 Actions Packages Projects Releases 12 Wiki Activity

Explore: run the credit ingest against the real dump and publish an artifact that carries credits #88

New Issue
Open
opened 2026-08-18 20:05:28 +00:00 by yonlu · 1 comment
yonlu commented 2026-08-18 20:05:28 +00:00
Owner
Copy Link
Copy Source

Multi-artist credits are built, shipped in code, and render for nobody, because no published artifact carries the credit tables. Every credit still falls back to its single link until an index build that includes the credit pass runs and is exported.

Converted from .planning/plans/active/015-multi-artist-credits.md, which is otherwise complete and moves to completed/.

What is already done

  • Phase 1 — backend/explore/dumpcredits.go + dumpcreditswrite.go, wired into dumpimport.go's run behind its own credits_import_done marker.
  • Phase 2 — cmd/indexexport writes artist_credit_part / artist_credit_ref; artifactimport.go reads them behind artifactHasCredits().
  • Phase 4 — explore.GetCredits reads the catalog tables keyed on the recording MBID, which both a catalog row and a local file carry, so one binding serves Explore and the library's own lists. All ten artist-link call sites render credits today.

What is left

Run the ingest against the real dump and publish the result. Everything is covered by tests over a synthetic tar, which cannot catch a surprise in the other ~35M rows.

The pass reads mbdump.tar.bz2 — 7.1 GB, roughly 13.7 minutes in pure-Go bzip2 — and depends on its members being alphabetical, which is what lets one pass resolve an entity's credit without buffering 35M recordings. It runs on every import mode, because a complete import means refresh, which never enters the importer at all, and it reports whether it populated anything so changed republishes the artifact.

Two things to watch on the first real run:

  • Column layouts. artist, artist_credit, artist_credit_name and recording were each read out of the real 20260815 export and are trusted. release_group is the one layout still taken on trust — it is assumed to share recording's first four columns. ErrDumpShape is what turns a wrong guess into a loud failure rather than a quietly wrong catalog, so a failure here is the system working.
  • The artifact must stay readable by older builds. artifactHasCredits() is the probe that makes a credit-less artifact still a valid catalog; the same shape as artifactStoresText and artifactHasTotals beside it. Adding the tables to the importer's SELECT without asking first is how a published artifact starts failing with no such column.

Why it is worth the run

Measured 2026-08-16 against a real 26,069-file library: 13% of recordings are multi-artist upstream, while only 0.86% of files carry a structured multi-artist tag — mp3 carries zero across 19,840 files. So this cannot be a tag-parsing feature, and the catalog is the only source that has the answer. Coverage is not a concern: 24,808 of 24,885 distinct recording MBIDs (99.7%) already have an explore_index row.

Done when

An artifact carrying artist_credit_part / artist_credit_ref is published, and a track credited to more than one artist renders every artist as its own link.

Multi-artist credits are built, shipped in code, and **render for nobody**, because no published artifact carries the credit tables. Every credit still falls back to its single link until an index build that includes the credit pass runs and is exported. Converted from `.planning/plans/active/015-multi-artist-credits.md`, which is otherwise complete and moves to `completed/`. ## What is already done - **Phase 1** — `backend/explore/dumpcredits.go` + `dumpcreditswrite.go`, wired into `dumpimport.go`'s `run` behind its own `credits_import_done` marker. - **Phase 2** — `cmd/indexexport` writes `artist_credit_part` / `artist_credit_ref`; `artifactimport.go` reads them behind `artifactHasCredits()`. - **Phase 4** — `explore.GetCredits` reads the catalog tables keyed on the *recording* MBID, which both a catalog row and a local file carry, so one binding serves Explore and the library's own lists. All ten artist-link call sites render credits today. ## What is left **Run the ingest against the real dump and publish the result.** Everything is covered by tests over a synthetic tar, which cannot catch a surprise in the other ~35M rows. The pass reads `mbdump.tar.bz2` — 7.1 GB, roughly 13.7 minutes in pure-Go bzip2 — and depends on its members being alphabetical, which is what lets one pass resolve an entity's credit without buffering 35M recordings. It runs on **every** import mode, because a complete import means `refresh`, which never enters the importer at all, and it reports whether it populated anything so `changed` republishes the artifact. Two things to watch on the first real run: - **Column layouts.** `artist`, `artist_credit`, `artist_credit_name` and `recording` were each read out of the real 20260815 export and are trusted. **`release_group` is the one layout still taken on trust** — it is assumed to share `recording`'s first four columns. `ErrDumpShape` is what turns a wrong guess into a loud failure rather than a quietly wrong catalog, so a failure here is the system working. - **The artifact must stay readable by older builds.** `artifactHasCredits()` is the probe that makes a credit-less artifact still a valid catalog; the same shape as `artifactStoresText` and `artifactHasTotals` beside it. Adding the tables to the importer's SELECT without asking first is how a published artifact starts failing with `no such column`. ## Why it is worth the run Measured 2026-08-16 against a real 26,069-file library: **13%** of recordings are multi-artist upstream, while only **0.86%** of files carry a structured multi-artist tag — mp3 carries *zero* across 19,840 files. So this cannot be a tag-parsing feature, and the catalog is the only source that has the answer. Coverage is not a concern: 24,808 of 24,885 distinct recording MBIDs (99.7%) already have an `explore_index` row. ## Done when An artifact carrying `artist_credit_part` / `artist_credit_ref` is published, and a track credited to more than one artist renders every artist as its own link.
logan commented 2026-08-19 00:38:12 +00:00
Collaborator
Copy Link
Copy Source

The republish owes a second column, not just the credit tables.

Measured against the live artifact today while confirming #7's Direction part 2:

$ curl -sSI .../generic/yellowjacket-core-index/latest/core-index.db.zst
last-modified: Mon, 10 Aug 2026 04:38:16 GMT

$ sqlite3 core-index.db \
    "SELECT COUNT(*) FROM pragma_table_info('explore_index') WHERE name='total_tracks';"
0
$ sqlite3 core-index.db "SELECT COUNT(*) FROM explore_index;"
1079667

total_tracks landed in the schema on 2026-08-16 and the published artifact is from 08-10, so it is in exactly the position the credit tables are: shipped in code, read behind a probe (artifactHasTotals()), and backed by no data for anybody.

Two consequences worth having written down here, since this issue is what closes both:

  • The "Done when" is really two facts. A track credited to more than one artist renders every artist as a link, and SELECT COUNT(*) FROM pragma_table_info('explore_index') WHERE name='total_tracks' returns 1 on the published file. It costs one query to check and it is the kind of thing that is assumed rather than checked.
  • artifactHasTotals() is now doing its job for real, not hypothetically — the current artifact is precisely the "built before the column existed" case it exists for, and it is why Explore is not failing with no such column right now. Same for artifactHasCredits() after this.

What that costs today: completenessAnswer()'s catalog fallback returns 0 for every user, so an album whose tags declare no total wears the plain tick with no second opinion available. #16 fixed the tag side; this is the other side.

**The republish owes a second column, not just the credit tables.** Measured against the live artifact today while confirming #7's Direction part 2: ``` $ curl -sSI .../generic/yellowjacket-core-index/latest/core-index.db.zst last-modified: Mon, 10 Aug 2026 04:38:16 GMT $ sqlite3 core-index.db \ "SELECT COUNT(*) FROM pragma_table_info('explore_index') WHERE name='total_tracks';" 0 $ sqlite3 core-index.db "SELECT COUNT(*) FROM explore_index;" 1079667 ``` `total_tracks` landed in the schema on 2026-08-16 and the published artifact is from 08-10, so it is in exactly the position the credit tables are: shipped in code, read behind a probe (`artifactHasTotals()`), and backed by no data for anybody. Two consequences worth having written down here, since this issue is what closes both: - **The "Done when" is really two facts.** A track credited to more than one artist renders every artist as a link, *and* `SELECT COUNT(*) FROM pragma_table_info('explore_index') WHERE name='total_tracks'` returns 1 on the published file. It costs one query to check and it is the kind of thing that is assumed rather than checked. - **`artifactHasTotals()` is now doing its job for real**, not hypothetically — the current artifact is precisely the "built before the column existed" case it exists for, and it is why Explore is not failing with `no such column` right now. Same for `artifactHasCredits()` after this. What that costs today: `completenessAnswer()`'s catalog fallback returns 0 for every user, so an album whose *tags* declare no total wears the plain tick with no second opinion available. #16 fixed the tag side; this is the other side.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Area/Design
Area/Downloads
Area/Explore
Area/Library-UI
Area/Metadata
Area/Packaging
Area/Player
Area/Queue
Area/Settings
Area/Shell-Nav
Compat/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
 Platform/Android
Platform/Desktop
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Medium
3
The priority is medium
Priority
Low
4
The priority is low
Reviewed
Confirmed
1
Issue has been confirmed
Reviewed
Duplicate
2
This issue or pull request already exists
Reviewed
Invalid
3
Invalid issue
Reviewed
Won't Fix
3
This issue won't be fixed
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status
In Progress
Somebody is actively working on this right now
No labels Area/Explore Area/Metadata Kind/Feature Platform/Android Platform/Desktop
Priority
Medium
3
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
logan yonlu
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Blocks
#89 Explore: materialise multi-artist credits locally so they survive an absent catalog
yonlu/yellowjacket
Reference: yonlu/yellowjacket#88