yonluandClaude Sonnet 5
e190fd75b9
Ships the fresh-start schema cleanup: rebuilt explore catalog index pipeline (dump import, artifact fetch/build, incremental listen-count refresh), a new download subsystem (Lidarr/Prowlarr/qBittorrent/SABnzbd/ slskd/yt-dlp providers, staging, reconciliation, wanted list), and the supporting schema/query/store changes across backend and frontend. Also includes two smaller follow-ups: bump the central index's rebuild-after cadence from 90 to 180 days, and remove the Explore "library only" online/offline toggle entirely (frontend-only, no backend counterpart) rather than carry unused UI/state. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y2Agd9af5hE7qzti2ackiS
141 lines
6.5 KiB
Markdown
141 lines
6.5 KiB
Markdown
# Notes
|
||
|
||
Gotchas, measured facts, and things already considered and rejected.
|
||
Measurements carry the date they were taken — several of these are
|
||
properties of someone else's server and can change.
|
||
|
||
## MetaBrainz caps a client at ~2 MB/s (measured 2026-07-29)
|
||
|
||
`data.metabrainz.org` serves a single client at roughly 2.1 MB/s, and
|
||
**concurrency does not help**: one Range stream and four concurrent
|
||
lanes delivered 32 MB at 2,111,195 B/s and 2,209,000 B/s respectively,
|
||
while the same machine pulled 66.9 MB/s from a CDN. One of the four
|
||
lanes starved to 0.5 MB/s. The lanes divide a fixed cap; they do not
|
||
raise it.
|
||
|
||
Consequences:
|
||
|
||
- No client-side concurrency change will speed up a dump download.
|
||
Pushing harder earns 503s (the reason `dumpLanes` is 4).
|
||
- Stage 1 of a full import costs ~11.8 h at best (89 GB after column
|
||
projection). Before projection it was 205 GB — about 27 h.
|
||
|
||
This is the entire reason the catalog is built centrally and shipped as
|
||
an artifact rather than derived per install.
|
||
|
||
## Further stage-1 reductions, not yet taken
|
||
|
||
Both are CI-side options; neither is safe as a silent client default
|
||
because each changes *what gets counted*.
|
||
|
||
- **Project `recording_mbid` only** (24.1% of row-group bytes instead of
|
||
43.4%): ~49 GB, ~6.5 h. `canonical_musicbrainz_data.csv` already
|
||
carries `recording_mbid`, `release_mbid`, `artist_mbids` and
|
||
`release_group_mbid`, so release/RG/artist counts can be rolled up
|
||
locally. Cost: listens with no recording MBID are dropped, and artist
|
||
totals become "sum of their recordings" rather than direct attribution.
|
||
- **Stride-sample members** (1-in-4): ~12 GB, ~1.6 h. The dump is flat
|
||
numbered members (`0.parquet`, …). Sampling is viable because the
|
||
counts only feed a *ranking* for a top-N cut. Must be a stride, never
|
||
a prefix — if members are time-ordered a prefix biases hard toward one
|
||
era.
|
||
|
||
## Incremental dump retention is 30 days (measured 2026-07-29)
|
||
|
||
The incremental directory held 30 dumps (series 2579–2610), and full
|
||
dumps land roughly monthly. An artifact older than ~30 days cannot be
|
||
topped up: the dailies bridging the gap are gone. That is a permanent
|
||
undercount of that window, not corruption — but it pins the artifact
|
||
republish cadence at monthly.
|
||
|
||
## Anonymous package download is UNVERIFIED
|
||
|
||
The client fetches the artifact from a fixed `latest` URL because Gitea's
|
||
package *listing* API requires a token while a plain file GET appears not
|
||
to — a probe of the not-yet-published artifact returned 404 rather than
|
||
401. **That is suggestive, not proof.** No artifact has been published
|
||
yet to test against. Confirm before relying on it.
|
||
|
||
Also worth deciding deliberately: every install pulling from a personal
|
||
Gitea makes its bandwidth and uptime a user-facing dependency.
|
||
|
||
## No migration chain
|
||
|
||
`applySchema` creates the whole schema from `sql/schemas/*.sql` on every
|
||
open; all DDL is `IF NOT EXISTS`. A database written by an older build is
|
||
not supported and there is no upgrade path by design.
|
||
|
||
Two things this replaced, worth not reintroducing:
|
||
|
||
- The 48-step chain was ~3,700 of `database.go`'s 4,061 lines, plus
|
||
helpers that existed only to serve it (`backupDatabase`,
|
||
`readLibraryDirFromTOML`, `isDuplicateColumnErr`, …).
|
||
- `sql/schemas/` had drifted badly from the real schema — it still
|
||
described a `genre_recordings` table that migrations had renamed, and
|
||
omitted `explore_index`, `http_cache`, `artist_images`,
|
||
`similar_artist_map`, `release_to_rg`, `lyrics_index` and
|
||
`artist_metadata` entirely. sqlc reads that directory, so it had been
|
||
generating against a stale schema and silently missed columns such as
|
||
`audio_files.modified_at`.
|
||
|
||
**When regenerating schema files from a live database, remember the seed
|
||
rows.** `file_types` (the four supported extensions), `player_state` and
|
||
`queue` each carry `INSERT OR IGNORE` rows that `sqlite_master` does not
|
||
contain. Dropping them breaks every audio-file foreign key.
|
||
|
||
## ANALYZE runs after the catalog merge, not at schema creation
|
||
|
||
The old migration 45 ran `ANALYZE` once. With the migration chain gone
|
||
there is no equivalent moment — an empty database has nothing to measure
|
||
— so it runs at the end of the artifact import instead
|
||
(`SearchIndex.analyzeIndex`). Without current statistics the planner
|
||
mis-estimates the partial expression indexes on `explore_index`
|
||
(`idx_explore_title_lower`, `idx_explore_artist_lower`) and scans a
|
||
million rows for queries that should seek.
|
||
|
||
If another path ever populates the catalog, it needs the same call.
|
||
|
||
## Writers, not readers, are responsible for name quality
|
||
|
||
`resolveArtistName` falls back to returning the artist MBID when it
|
||
cannot find a name. That is fine for a one-off render but must never be
|
||
persisted — an MBID stored as a title is unsearchable and shows as a
|
||
UUID in the UI.
|
||
|
||
This used to be defended at every read (`title != mbid` predicates) and
|
||
in the upsert's conflict rules. Those defenses are gone; `AddFromCache`
|
||
now refuses to write a name equal to the MBID and lets the upsert's
|
||
"non-empty wins" rule fill it in when a real name arrives.
|
||
`TestAddFromCacheNeverStoresMBIDAsName` guards this.
|
||
|
||
## Attached databases are invisible to the read pool
|
||
|
||
`database.DB` holds two handles: a single-writer connection and a
|
||
separate query-only pool. `ATTACH` binds to one connection, so anything
|
||
touching an attached database must use `ExecContext`/`QueryRowWriter`
|
||
(the writer) — `QueryContext` routes to the pool, where the attachment
|
||
does not exist and the query fails with "no such table".
|
||
|
||
## FTS triggers are defined in Go, not in the schema
|
||
|
||
`explore_index`'s three FTS sync triggers live in
|
||
`exploreIndexFTSTriggers` in `database.go` rather than in
|
||
`sql/schemas/explore_index.sql`, because the bulk-load path drops and
|
||
recreates them (`SuspendExploreIndexFTS`). Defining them in both places
|
||
would be two copies free to drift.
|
||
|
||
Bulk loads must suspend them: measured on a real import, assembly runs at
|
||
~31 rows/s with the triggers attached and ~4,700 rows/s without.
|
||
|
||
## Explore "library only" toggle was removed (2026-08-06)
|
||
|
||
The Explore UI used to have a "library only" mode toggle
|
||
(`frontend/src/store/explore-settings.ts`, `explore:libraryOnly` in
|
||
localStorage) that filtered the Explore UI to owned content only. It was
|
||
removed outright — the app now always shows full (network-enriched)
|
||
Explore data. If offline/library-only mode is wanted again, it should be
|
||
built from scratch rather than restored; the old implementation gated
|
||
several component code paths in ad hoc ways. A separate
|
||
`deep_catalog_enabled` backend flag briefly existed for the same idea and
|
||
was removed earlier, when the dump importer left the app binary.
|