yonlu/yellowjacket
1
1
Fork 0
Code Issues 52 Pull Requests 4 Actions Packages Projects Releases 12 Wiki Activity

feat: data lifecycle rewrite, download clients, wanted list, and central catalog index
Build & publish Arch package / arch-package (push) Successful in 2m12s
Details
Search index maintenance / maintain-index (push) Successful in 2h22m28s
Details

Browse Source 
Ships the fresh-start schema cleanup: rebuilt explore catalog index
pipeline (dump import, artifact fetch/build, incremental listen-count
refresh), a new download subsystem (Lidarr/Prowlarr/qBittorrent/SABnzbd/
slskd/yt-dlp providers, staging, reconciliation, wanted list), and the
supporting schema/query/store changes across backend and frontend.

Also includes two smaller follow-ups: bump the central index's
rebuild-after cadence from 90 to 180 days, and remove the Explore
"library only" online/offline toggle entirely (frontend-only, no
backend counterpart) rather than carry unused UI/state.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y2Agd9af5hE7qzti2ackiS
This commit is contained in:
yonlu
2026-08-06 17:12:01 -04:00
co-authored by Claude Sonnet 5
parent d0d86f85d5
commit e190fd75b9
165 changed files with 31088 additions and 5192 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.planning/plans/completed/001-ship-core-index.md
+379
View File
@@ -0,0 +1,379 @@
# 001 — Ship a prebuilt "core" explore index
**Status:** complete
**Branch:** cleanup/fresh-start-schema
**Created:** 2026-07-25
**Completed:** 2026-07-30
## Outcome
A fresh install downloads a 70.6 MB artifact and merges 1,076,133 rows
in ~43 s, instead of streaming 205 GB over ~27 h. The dump importer that
produces the artifact left the app binary entirely — it is behind the
`indexbuild` build tag and runs only in CI.
Phase 5 landed differently than planned: rather than a user-facing
setting gating the deep import, the deep import is simply not in the
app. `deep_catalog_enabled` existed briefly and was removed with it.
Two things remain unverified or undone, both recorded in
`.planning/NOTES.md`: anonymous package download on git.ljones.me has
not been confirmed against a real published artifact, and installs whose
index was built by older code (no `listens_applied_series`) have no
rescue path — though with no migration chain, those databases are now
unsupported anyway.
## Problem
A fresh install has no explore index. `StartIndexBuild()` is called
unconditionally from two places in `app.go`, and `runDumpBuild` then
downloads gigabytes from `data.metabrainz.org` before Explore can return
anything beyond the user's own library:
| Stage | Source | Cost |
|---|---|---|
| Listen Counts | ListenBrainz spark full listens dump | **~205 GB streamed** — see below |
| Catalog Import | MusicBrainz canonical dump (~2 GB `.tar.zst`) | scan ~30M CSV rows, assemble to budget |
| Metadata Patch | MB/LB API | rate-limited at 3 req/s |
| Listener Counts | LB API | rate-limited |
Measured 2026-07-25 against the live dump
(`listenbrainz-spark-dump-2593-20260712-000004-full.tar`):
```
content-length: 205073162240 # 205 GB
accept-ranges: bytes
```
The stage-1 reader skips non-`.parquet` tar members
(`dumpcounts.go:317`), but a tar stream has no seek — skipped bytes
still transit the wire. **So a first run on a fresh install pulls
~205 GB.** Little of it touches disk (the counts map and checkpoint do,
not the dump), but the bandwidth is real and it is per-user.
Consequences today:
- Every install pulls ~205 GB to derive a catalog that is **identical
for everyone**. On a metered or slow connection this is untenable, and
it is unconditional on first run.
- **It refuses to start without 6 GB free** (`dumpMinStartFreeBytes`),
and aborts below 2 GB (`dumpAbortFreeBytes`). This is what breaks
`make fresh-install` on a tmpfs `/tmp`.
- First-run Explore is empty for the length of the import.
The catalog half is **the same for everyone**. Only the local half
(`PopulateLocalCrossReferences`, `BackfillLibraryDiscographies`) is
per-user. Deriving the shared half on each machine is the waste this
plan removes.
## Goal
Ship a prebuilt core index so a fresh install has a usable Explore
immediately, and the runtime build collapses to the local half plus
incremental refresh. The full dump import becomes an opt-in "deep
catalog" upgrade rather than a prerequisite.
## Sizing evidence
Measured 2026-07-25 with a synthetic harness against the real schema and
migrations (2.15M-row full run exceeded a 15-minute budget, so this is a
200K-row calibration, `VACUUM`ed):
| Metric | Value |
|---|---|
| 200,000 rows, with FTS | 85.2 MB |
| Cost per row | ~426 B |
| zstd -19 | 29.6 MB (2.9x) |
Extrapolating to the current budgets (`keepRecordings` 1.5M +
`keepReleaseGroup` 400K + `keepArtists` 250K = 2.15M rows):
| Tier | Rows | On disk | zstd -19 |
|---|---|---|---|
| Full budget | 2.15M | **~900 MB** | ~310 MB |
| Core (proposed) | 500K | ~210 MB | **~72 MB** |
| Minimal | 250K | ~105 MB | ~36 MB |
**This corrects an earlier figure.** A ~93 MB index was recorded in the
2026-07-16 audit note; that measured the *legacy tier-crawl* index, not
the dump-built one. The dump build targets an order of magnitude more
rows. Shipping the full index is not viable as a casual download —
which is exactly why this plan is scoped to a *core* subset.
⚠️ Two caveats on these numbers:
- The harness used a 14-word vocabulary, so its FTS measured only 7% of
total size. Real titles have a far larger vocabulary and the real FTS
share will be materially higher. **Treat the totals as a floor.**
- Row width was estimated from the schema (3 UUIDs at 36 chars dominate);
`aliases` was left empty and is populated for real artists.
Re-measure against a genuine dump-built index before committing to a
tier size.
## What "core" should mean
`dumpcatalog.go` already has graded per-artist coverage (S2) —
`perArtistArtistBudget = 10_000` split into tiers A/B/C with per-tier
track and release-group caps. The core index should reuse that machinery
rather than invent a second notion of importance:
- **Artists:** top ~50K by listen count.
- **Release groups + recordings:** the S2 per-artist slice for those
artists (tier A/B/C caps as they stand).
- **Excluded:** the global long tail below the per-artist selection.
Anything not covered still works — it just resolves through the existing
lazy paths (`EnsureArtistDiscography`, `AddFromCache`), which is the
behaviour non-covered artists already get today.
## Distribution: download on first run, not `go:embed`
**Both packaging paths build from source** — the Homebrew formula builds
from a release tarball, the Arch `PKGBUILD` clones the tag. So:
- Committing the artifact to git bloats the repo and every source tarball.
- `go:embed` makes a from-source build require the artifact at build
time, so source builds would have to download it anyway — and
`build-prod` runs UPX over the binary, which would be pathological
with a 70 MB+ embedded blob.
So "ship with the app" should mean **fetch a prebuilt artifact on first
run** from a versioned URL. CI already publishes binary packages to the
Gitea package registry (`.gitea/workflows/arch-package.yml`), so there is
an existing place to host it.
Import path: download `.zst` → decompress → `ATTACH` → `INSERT INTO
explore_index SELECT ...` through the **existing** `upsertBatch` conflict
rules, which already do the right thing (non-empty wins, highest
popularity wins, never clobber a good value with an empty one).
## Artifact contents
Ship the global catalog columns only. These are **per-user** and must be
zeroed in the artifact, then recomputed locally by
`PopulateLocalCrossReferences`:
- `in_library`, `is_similar`
- `local_artist_id`, `local_release_group_id`, `local_recording_id`
`discog_fetched` should ship as `1` for artists whose S2 slice is
included, so the backfill doesn't redundantly re-fetch them.
Also decide per-table whether to include: `similar_artist_map`,
`artist_metadata`, `release_to_rg`. `release_to_rg` in particular may
rival the index in size — measure before including.
**Resolved: the artifact ships no FTS.** Rows are inserted into the
client's own `explore_index`, whose `AFTER INSERT` trigger populates
`explore_index_fts` as a side effect — so shipping a search index would
be pure redundant weight. `cmd/indexexport` builds the artifact without
FTS or triggers accordingly.
## Update strategy
- **Popularity drift** — `dumpincremental.go` already implements
incremental listens-dump refresh (`RefreshListenCounts`, weekly
cadence). It applies unchanged on top of a shipped baseline, provided
`listens_applied_series` is stamped in the artifact so deltas resume
from the right point.
- **Catalog additions** — new releases arrive via the existing lazy
per-artist fetches. A refreshed artifact per app release is enough;
no separate cadence needed.
- **Schema changes** — `schema_version` exists on `explore_index` but is
noted as dead in the audit. Either wire it up or version the artifact
filename against the migration number, so an old artifact can't be
imported into a newer schema.
## Build pipeline: build and cache in Gitea CI
The import is unusually well suited to running as a **series of
time-boxed CI jobs against a persistent cache**, because the resumability
already exists:
- Stage 1 streams over a `resumableReader` that reconnects with HTTP
`Range` requests, and the live dump advertises `accept-ranges: bytes`.
- `counts.bin` checkpoints `Offset` (absolute byte position) and
`MemberIdx`, and the applier merges results **in member order** so
"every checkpoint is a contiguous prefix of the stream"
(`dumpcounts.go`).
- Stage 2's canonical scan is deliberately restartable wholesale — "cheap
enough to simply restart after an interruption" (`dumpcatalog.go`).
So a job that hits a runner time limit resumes at its exact byte offset
on the next run. **No single multi-hour job is required** — schedule
N bounded runs and let them converge.
What it needs:
1. **A persistent volume for `explore-staging/` + the DB.** `act_runner`
uses the Docker backend and job containers are ephemeral, so bind-mount
a host path (or a named Docker volume) and point `YJ_HOME` at it.
Prefer this over the Actions cache — cache entries are size-capped and
awkward at GB scale, and this is a self-hosted runner anyway.
2. **A headless entrypoint** — currently the import only runs from the
app lifecycle (`StartIndexBuild` via `OnDomReady`). This is a real gap,
but a small one: `NewSearchIndex(db, lb, artistImg, logger)` takes no
Wails dependency, and the single `runtime.EventsEmit` in
`searchindex.go` sits inside `emitStatus`, which already early-returns
when `runtimeCtx == nil`. A `cmd/indexbuild` that opens the DB and
calls `StartBuild(context.Background())` — never `SetContext` — should
work. Verify `scheduleChampionRebuild` in the `StartBuild` defer is
also Wails-free.
3. **Triggers.** `indexbuild` decides its own mode from index state, so
every trigger runs the same command: push to `main` and a weekly cron
both land on a cheap refresh (which no-ops when nothing new is
published), and the 3-month rebuild fires when the command notices the
import has aged out.
Then export: subset to core, zero the personal columns, stamp
`dump_import_done` / `listens_applied_series` / schema version, `VACUUM`,
`zstd -19`, checksum, publish to the Gitea package registry (the Arch
workflow already authenticates against it with `PACKAGE_TOKEN`).
**Be a good citizen about the 205 GB.** Rebuild on the dump cadence
(the audit notes a 90-day re-import cadence), never per-commit. Once a
baseline exists, the ~180 MB daily incremental dumps already wired in
`dumpincremental.go` keep popularity fresh — so the 205 GB is genuinely
one-time per rebuild, not per refresh. Also check the runner's own
egress if it is self-hosted on a home connection.
## Licensing
- MusicBrainz canonical dump is **CC0** — redistribution fine.
- ListenBrainz-derived listen counts need their dump licence checked
before redistribution, plus attribution in-app either way.
- Note the derived counts already differ from LB API values (no MLHD+
history) — a known, accepted divergence, but worth stating wherever
the numbers are surfaced.
## Risks
- **Artifact staleness vs app version** — a user on an old release gets
an old catalog. Mitigated by incremental refresh + lazy fetches.
- **Download failure / offline install** — must degrade to today's
behaviour (local library search), not a broken Explore. The failure is
now visible in the Jobs panel, which helps.
- **Users who want the full catalog** — keep the existing dump import as
an explicit opt-in, gated behind a setting. Note that no such setting
exists today: `StartIndexBuild()` is unconditional, and Library Only
mode is frontend-`localStorage` only with no backend wiring.
## Phasing
1. ✅ **Headless entrypoint.** `cmd/indexbuild` — resumable, budgeted
(`-budget 3h`), signal-aware, exit 3 = "more work remains". Verified
to run without Wails; builds with `CGO_ENABLED=0` and no build tags.
2. ✅ **Export tooling.** `cmd/indexexport` — top-N artists plus a
per-artist window of their release groups and recordings, personal
columns dropped, metadata stamped, vacuumed. Verified against a
synthetic index: no personal columns leak, no orphaned rows, caps
respected.
3. ✅ **One real build.** Superseded by a real dump-built index that
already existed on the dev machine (`dump_import_done` 2026-07-17).
Measured 2026-07-29 — these replace every extrapolation above:
| | rows | on disk |
|---|---|---|
| `explore_index` | 2,052,168 (227,359 artists / 400,675 RGs / 1,424,134 recordings) | 383 MB |
| its indexes | | 395 MB |
| FTS | | 80 MB |
187 B/row for the shippable table, 418 B/row all-in — so the ~900 MB
full-budget estimate was right. Two real exports:
| tier | rows | artifact | zstd -19 |
|---|---|---|---|
| 50K artists (default) | 1,076,133 | 191.5 MB | **70.6 MB** |
| 25K artists / 10 RG / 20 rec | 620,973 | 110.6 MB | **37.8 MB** |
`release_to_rg` was empty in that index — it predates the code that
persists it — so its size is still unmeasured.
4. ✅ **Import path.** `backend/explore/artifactfetch.go` (download,
Range-resume, sha256, zstd) and `artifactimport.go` (validate, ATTACH,
batched merge, FTS rebuild, meta stamping). Reported in the Jobs panel
under its own two stages. Measured end to end on the real 50K-artist
artifact against a disk-backed DB: **1,076,133 rows merged in 43.2s**
(24,900 rows/s), yielding a 455 MB `yj.db`, FTS populated and
searchable. In-memory the same merge runs in 28.3s.
5. ✅ **Gate the dump build.** `deep_catalog_enabled` in
`explore_index_meta` (beside `index_build_paused` — it is build state,
read at one decision point). Off by default; exposed as
`DeepCatalogEnabled` / `SetDeepCatalogEnabled` on the explore Service.
An interrupted dump import resumes regardless of the setting, so the
gate never discards a checkpoint that already cost hours.
## Measured 2026-07-29: why the client cannot fix this itself
`data.metabrainz.org` caps a client at ~2.1 MB/s. One Range stream and
four concurrent Range lanes both delivered 32 MB at the same aggregate
rate (2,111,195 B/s vs 2,209,000 B/s) while the same machine pulled
66.9 MB/s from a CDN. **Parallelism buys nothing** — the four lanes just
divide the same cap, and one of them starved to 0.5 MB/s.
So stage 1 costs, unavoidably:
| | bytes | wall clock |
|---|---|---|
| Whole tar (what shipped before column projection) | 205 GB | ~27 h |
| Column projection, 3 columns (43.4%) | 89 GB | ~11.8 h |
| `recording_mbid` only (24.1%), rolled up via canonical | 49 GB | ~6.5 h |
| + 1-in-4 member stride sample | 12 GB | ~1.6 h |
The last two are CI-side options, not client defaults: recording-only
drops listens carrying no recording MBID and re-derives artist totals as
a sum over recordings, and sampling trades exact counts for a ranking.
Both are only safe because the selection they feed is a top-N cut.
## Distribution: the "latest" version trick
The client cannot enumerate package versions — Gitea's package listing
API requires a token, while an anonymous file GET does not (a probe of a
non-existent artifact returns 404, not 401). So `index-artifact.yml`
publishes each artifact twice: under a dated version for history, and
under a fixed `latest` version that the client fetches from a
predictable URL. Generic packages reject overwriting an existing
filename, so `latest` is DELETEd before each rewrite.
⚠️ **Unverified:** that anonymous package *download* is actually enabled
on git.ljones.me. The 404-vs-401 probe is suggestive, not proof — no
artifact has been published yet to test against. Confirm before relying
on it, and note that every install pulling from a personal Gitea makes
its bandwidth and uptime a user-facing dependency.
## Incremental retention bounds artifact staleness
The incremental dump directory holds 30 dumps (series 2579–2610 as of
2026-07-29) and full dumps land roughly monthly. An artifact older than
~30 days therefore cannot be topped up: the dailies bridging the gap are
gone. That is a permanent undercount of that window's listens, not
corruption — but it pins the republish cadence at monthly.
## Upgrade path for indexes built by older code
The dev machine's index has `dump_import_done` set but **no**
`listens_applied_series` and an empty `release_to_rg`, because it was
built before the code that writes them. That combination is a dead end:
`RefreshListenCounts` bails with "no baseline series recorded", and
`runDumpBuild` short-circuits on the done marker, so popularity can
never update again. Current code writes both, so this affects only
pre-existing installs — but the artifact import is the natural place to
rescue them, since merging one stamps a fresh baseline series.
6. ✅ **CI wiring.** `.gitea/workflows/index-artifact.yml` — push +
weekly cron + manual, concurrency-guarded, publishes only when
`complete && changed` so identical artifacts don't accumulate.
Runner-side prerequisites are in place (cache dir + `valid_volumes`
on the VPS runner).
Step 3 is the gate on everything downstream — and it is worth doing
regardless of whether the artifact ever ships, since it is the only way
to get real numbers for the index.
## Related
- `backend/explore/dumpimport.go` — stage orchestration, disk floors
- `backend/explore/dumpcatalog.go` — budgets, S2 per-artist tiers
- `backend/explore/dumpincremental.go` — incremental refresh (update path)
- `backend/explore/searchindex.go` — `upsertBatch` conflict rules,
`PopulateLocalCrossReferences`
- Migration 26 in `backend/database/database.go` — `explore_index` schema
.planning/plans/completed/002-data-lifecycle-architecture.md
+155
View File
@@ -0,0 +1,155 @@
# 002 — Data lifecycle architecture
**Status:** completed (first tranche); follow-ups tracked below
**Branch:** main
**Created:** 2026-07-26
## Problem
An audit of asset and row cleanup found five leaks, four of which shared
one root cause: **deletion logic was hand-written per call site and lived
far from the thing being deleted.** `RemoveLibrary` knew about ten tables
because someone enumerated them once; migration 32 added an eleventh and
nothing noticed. Files written by `explore` had no cleanup counterpart
anywhere. A function that evicted expired cache rows was written and
never called.
Findings, in severity order:
1. **`RemoveLibrary` was broken for any scanned library.** `tagging_items`
holds `FOREIGN KEY(library_id) REFERENCES libraries(id)` with no
`ON DELETE` clause and was never cleared, so `DELETE FROM libraries`
failed with `FOREIGN KEY constraint failed (787)` and rolled back the
whole removal. Every scanned library has `tagging_items` rows (the
scan upserts one per album folder), so this fired on essentially every
real removal. `RemoveLibrary` had zero test coverage.
2. **Artist images were never deleted by anything.** No `os.Remove` in
`explore`, no `DELETE FROM artist_images` in the codebase. Unbounded
in the number of artists ever browsed in Explore, most of whom are not
in the library.
3. **Cover art size variants leaked on removal.** Only the base
`cover_art.file_path` was unlinked; the `_sm/_md/_lg` files beside it
are derived filenames, not rows, so three files per cover survived.
4. **`http_cache` was never pruned.** `Cache.Evict()` existed with no
callers. Reads filter on `expires_at`, so expired rows were inert but
accumulated for the life of the install.
5. **Cover-art proxy cache was never pruned.** No eviction, no size cap.
## Approach
Rather than patch five holes, classify the data so the *class* of bug
becomes hard to write. Everything persisted falls on two axes —
regenerability and cost of regeneration — which collapse to four kinds:
| Kind | Regenerable? | Deletion policy |
|---|---|---|
| **Owned** — projection of the user's files | Yes, by rescan | Follows the files |
| **Authored** — user-created, no other copy | **No** | Explicit user action only |
| **Derived** — computed from owned | Yes, cheaply | Free; must never block owned deletion |
| **Cache** — network or dump sourced | Yes, expensively | TTL/age eviction, never cascade |
The classification is not just vocabulary — it produces the right fix for
each finding. Finding 1 is derived data acting as a referential parent of
owned data, which the taxonomy makes categorically illegal. Finding 2 is
cache data that never needed owner-linked cleanup at all; it wants age
eviction. Finding 3 is derived data that must be swept against a live set
rather than tracked individually.
A Go interface was considered and rejected: the only polymorphic consumer
is the janitor, the substrates have nothing in common (SQL rows, an FTS
virtual table, a view, three directories of JPEGs, a 900 MB index), and
provenance is a static fact better enforced by package boundaries than by
methods an implementation may lie about. A declarative catalog gets the
same benefit for a tenth of the cost.
## What shipped
**`backend/datamap`** — the catalog. Every table, view, and asset
directory declared with its `Kind`, its `Lifetime` (`cascade`, `set-null`,
`swept`, `retained`), and a note explaining the classification. Plain data
with no service dependencies, so tests can assert it against a live
schema. FTS5 shadow tables resolve to their parent.
Tests that give it teeth (`backend/datamap/datamap_test.go`):
- `TestCatalogCoversSchema` — every table in `sqlite_master` is claimed by
exactly one entry. **A new table fails the build until somebody states
what it is and how it dies.**
- `TestCatalogHasNoStaleEntries` — the reverse, catching drift.
- `TestNoActionForeignKeysAreDeclaredSwept` — a `NO ACTION` foreign key
blocks its parent's deletion, so its table must declare `swept`. This is
the exact shape of finding 1, now caught at CI time.
- `TestLifetimesMatchSchema` — declared cascade/set-null must match what
SQLite actually enforces.
- `TestAuthoredCascadesAreDeliberate` — authored data is unrecoverable, so
a cascade onto it needs an explicit exemption.
**`backend/maintenance`** — the janitor. A registry of named jobs with
per-job minimum intervals, run at startup-idle and on a 6h tick. Policies
follow the taxonomy: derived data sweeps against a live set, cache data
ages out. Registered in one place (`app.go: startJanitor`) so the full set
of janitorial work is a single visible list.
Jobs: `http-cache-evict` (6h), `covers-sweep` (24h, live set from
`cover_art` expanded via `CoverArtFileSet`), `artist-images-sweep` (24h,
keeps art for library artists indefinitely, evicts browsed-artist art
after 90d), `cover-art-proxy-sweep` (24h, 30d age eviction).
The covers sweep refuses to act on an empty live set — that means the
query failed to see the table, not that every cover is garbage.
**Leak tests** (`backend/library/leak_test.go`) — driven by the catalog
rather than a hardcoded list, so new tables are covered the moment they
are catalogued:
- `TestRemoveLibraryLeavesNoOwnedOrDerivedRows` — removing the only
library leaves no owned or derived rows, except those in
`staleTolerated` with a written reason.
- `TestRemoveLibraryPreservesAuthoredData` — authored data survives.
- `TestSweptTablesAreActuallySwept` — a table declaring `swept` that
nothing sweeps is caught.
All three were verified to fail when the finding-1 fix is reverted.
**Fixes** — `tagging_items` cleared inside the removal transaction
(`crud.go` step 17); `CoverArtFileSet` expands originals to variants and
the legacy `_thumb` name; `Cache.Evict` logic moved into a registered job.
**Incidental:** `Library.emit` — `runtime.EventsEmit` calls `log.Fatalf`
on a context without a Wails runtime, which killed the test binary and
made the whole package untestable. All ten emits in the package now route
through a nil-safe helper. This also removes a real crash risk for
background workers that outlive their context.
## Follow-ups
**`audio_files` is a mixed-kind table.** `play_count`, `last_played`, and
`tag_status` are *authored* data living in an *owned* table. Orphan
cleanup treats the whole row as regenerable, which is why renaming a file
destroys its play count — the row is deleted and re-imported fresh. This
is the strongest argument for splitting authored per-track state into its
own table keyed by something more stable than a path. Related: an
audio-stream content hash (excluding tag blocks, so it survives
retagging) would let a rename be recognised as the same file. Deliberately
out of scope here; it is a schema change plus a rename-detection pass, not
a cleanup fix.
**Cascade adoption.** Fourteen of nineteen foreign keys are `NO ACTION`.
Converting them to `CASCADE` would delete a lot of hand-written orphan
sweeps, but SQLite cannot add `ON DELETE` via `ALTER TABLE` — each needs
the 12-step table rebuild. Note the ordering constraint: cascades delete
rows silently, so any code that collects file paths *before* deleting rows
(as `RemoveLibrary` does for cover art) breaks under cascade. Mark-and-
sweep must land first; the two compose, cascade plus path-collection does
not.
**Consolidate the ten orphan sweeps.** `DELETE ... WHERE id NOT IN (...)`
appears ten times across `crud.go`, `dbsync.go`, `smartplaylist.go`, and
`database.go`. One shared `sweepOrphans(tx)` would shrink the surface where
a new table can be forgotten. Worth doing opportunistically rather than as
a big-bang refactor.
**Storage settings pane.** The catalog knows every table and directory and
its kind; the janitor already computes bytes freed. A settings pane showing
per-kind disk usage with "clear cache" and "rebuild derived data" buttons
is now mostly a UI job.
.planning/plans/completed/003-download-clients.md
+279
View File
@@ -0,0 +1,279 @@
# 003 — Download clients
**Status:** implemented (v1); follow-ups tracked below
**Branch:** main
**Created:** 2026-07-27
## Problem
YellowJacket can find music (`explore`), identify it (`autotag`), and
manage it (`library`) — but it can't acquire it. The one gap between
"you're missing this album" and "you own this album" is filled today by
the user alt-tabbing to some other tool.
The naive fix is an HTTP client for Soulseek and a shell-out to yt-dlp.
That produces two bespoke code paths with duplicated queueing, retry,
staging and import logic, and a third service means a third copy. The
services users want to connect are also not the same *kind* of thing —
some search, some transfer bytes, some are entire automation systems we
delegate to — so a single `DownloadClient` interface would be a lie that
every adapter partially implements.
## The role decomposition
Every candidate integration fills one or two of three roles:
| Service | Searches | Transports | Delegates |
|---|---|---|---|
| slskd (Soulseek) | ✅ | ✅ | |
| yt-dlp | ✅ | ✅ | |
| Lidarr | | | ✅ |
| Prowlarr | ✅ | | |
| qBittorrent / Transmission | | ✅ | |
| SABnzbd / NZBGet | | ✅ | |
So: three small interfaces, not one big one. A provider implements
whichever it supports and declares that in a capability struct, the same
way `jobs.Caps` lets the frontend render controls without switching on
`Kind`.
```go
// Searcher turns a request into ranked candidates.
type Searcher interface {
Search(ctx context.Context, req Request) ([]Candidate, error)
}
// Transporter moves a candidate's bytes to a local staging directory.
type Transporter interface {
Grab(ctx context.Context, c Candidate, dst string, p ProgressFunc) (Result, error)
Cancel(ctx context.Context, grabID string) error
}
// Delegator hands the whole request to an external manager and
// reports back when files land.
type Delegator interface {
Request(ctx context.Context, req Request) (string, error)
Poll(ctx context.Context, externalID string) (DelegateStatus, error)
}
```
A `Provider` is the registry entry: identity, config, health check, caps,
plus whichever of the three it satisfies. Search-only providers
(Prowlarr) are paired with a transport at grab time by protocol match
(`torrent` → qBittorrent, `usenet` → SABnzbd); providers that do both
are self-pairing.
## v1 decisions (settled)
- **On-demand only.** User-initiated "find this album" from an Explore
artist/album page or a missing-album row. No wanted list, no artist
monitoring, no quality-cutoff upgrades. The queue and pipeline built
here are exactly what monitoring would later sit on top of — see
Deferred.
- **Soulseek via slskd's REST API**, not a native protocol client. Same
adapter shape as everything else, no wire protocol, no credentials in
our process, fully testable against an `httptest` server. A native
provider can slot in behind `Searcher`/`Transporter` later with no
pipeline changes.
- **Stage → autotag → import.** Downloads land in a staging directory,
are matched against the intended release with the existing `autotag`
scorer, tagged, then moved into the library and scanned. Never write
into the library root directly.
- **All four provider families in v1**, sequenced so each phase proves a
different role shape (see Phases).
## Pipeline
```
Request (MBID-anchored where possible)
└─> fan-out Search across enabled providers (per-provider timeout)
└─> merge + rank Candidates
└─> user picks (or auto-pick above confidence threshold)
└─> Grab into staging/<request-id>/
└─> verify (audio decodes, expected track count)
└─> autotag against the intended release
└─> tagwriter writes tags
└─> move into library layout
└─> targeted incremental scan
```
The `Request` should carry a release-group or release MBID whenever the
user started from an Explore page, because that anchor is what makes the
autotag step reliable instead of a second guess. Free-text requests are
supported but flagged lower-confidence, and never auto-pick.
Staging lives under the user data dir, not the library. Partial grabs are
resumable where the provider supports it and swept on startup where it
doesn't.
## Candidate ranking
Two independent scores, kept separate:
1. **Match confidence** — does this candidate contain the release the
user asked for? Reuse `autotag`'s distance/alignment machinery on the
candidate's *filenames* (Soulseek gives paths, not tags), against the
expected tracklist from the explore index.
2. **Source quality** — format (FLAC > V0 > 320 > lower), bitrate,
completeness (file count vs. expected track count), source health
(slskd queue length and upload slots; seeders for torrents), and a
user-set per-provider priority.
Ranking presents both, because they trade off — a perfectly-matched
128kbps rip should lose to a well-matched FLAC, and the user should be
able to see why. Reusing `autotag.ScoreBreakdown`'s "explain the ranking"
pattern here is deliberate.
## Persistence
New tables (migration TBD, next free number):
- `download_providers` — id, kind, name, enabled, priority, config blob
(JSON), `created_at`. Non-secret config only.
- `download_requests` — id, source (`explore-album`, `explore-artist`,
`manual`), release_mbid / release_group_mbid, free-text query,
requested_at, state, resolved_download_id.
- `download_items` — one row per grab attempt: request_id, provider_id,
candidate JSON, state, bytes/total, staging path, error, timestamps.
**Secrets** (slskd API key, Lidarr/Prowlarr API keys, qBittorrent
password) do not go in the TOML config or the DB in plaintext. Use the OS
keyring where available with a clearly-labelled encrypted-file fallback,
and never log a config value from a provider's secret field. Open
question below on the exact library.
## Jobs integration
Add `jobs.KindDownload`. One job per request (not per file), with
`Stages` for search → grab → import so the existing detail panel renders
the pipeline for free. `Caps{Cancellable: true}`; pausable only for
providers that can resume. Per-provider concurrency caps and a global
cap, both configurable — hammering a Soulseek peer with eight parallel
transfers gets you queued or banned.
## Frontend
- New `download-providers` section in `config-page` (HTMX + templ, same
as existing settings) for provider CRUD, test-connection, priority.
- New `download-picker` Lit component: the ranked-candidate dialog,
invoked from Explore album/artist pages and from a missing-album row.
- `download-store.ts` subscribing to the existing `JobsChanged` event —
no new event channel needed for progress.
## Phases
Each phase is independently shippable and proves a distinct role shape.
1. **Core.** Interfaces, registry, `Request`/`Candidate`/`Result` types,
staging dir, ranking, the stage→autotag→import tail, jobs wiring,
schema, secret storage. Ships with a fake provider and full test
coverage of the pipeline. No real network.
2. **yt-dlp.** Subprocess provider: search + transport, no server for the
user to run, so it's the fastest path to an end-to-end working
feature. Proves the local-subprocess shape (binary discovery,
version checks, stdout progress parsing, sandboxing the arg list).
3. **slskd.** Remote search + transport over REST. Proves the remote
HTTP shape and is the highest-value source. This is where filename-
based match confidence earns its keep.
4. **Lidarr.** Delegate. Proves the fire-and-poll shape, where we don't
own the transfer and the "import" step is really "detect what Lidarr
already imported and reconcile".
5. **Prowlarr + qBittorrent/SABnzbd.** Proves split search/transport
pairing — the one case where two providers cooperate on a single
request.
## Risks and constraints
- **No bundled credentials, no default-on providers, no preconfigured
indexers.** Every provider is off until the user configures it. The
app ships the ability to connect to services the user already runs.
- **yt-dlp is a moving target.** Pin a minimum version, check it at
provider-enable time, and fail with a clear message rather than
parsing garbage output.
- **Filename-only matching is genuinely hard.** Soulseek results are
`\Music\Album (1997) [FLAC]\01 - Track.flac` at best. Budget real
effort for the path-parsing heuristics; `autotag/normalize.go` is the
starting point.
- **Partial and failed grabs must never reach the library.** The import
step is the only writer into library paths, and it runs after
verification. Staging sweep on startup.
- **Tests must not hit the network.** `httptest` servers for slskd/
Lidarr/Prowlarr, a stub binary for yt-dlp.
## Deferred
- Wanted list with background retry (the natural next plan).
- Artist monitoring + auto-grab of new releases — cheap once the wanted
list exists, because `explore`'s dump index already knows the full
discography and `library` already knows what's owned.
- Quality profiles and upgrade-if-better.
- Native Soulseek protocol client.
- Transmission/Deluge/NZBGet (same shape as their shipped siblings —
add on demand).
- Internet Archive / Bandcamp-collection providers: cheap REST adapters,
worth adding once the core is proven.
## Resolved questions
1. **Secret storage.** No keyring dependency was added. Credentials go
in a 0600 JSON file in the user data directory (`download-secrets.json`),
keyed by provider row ID. This is deliberately *not* encryption — a
key stored beside the data it unlocks protects nothing, and claiming
otherwise would be worse than being clear about it. What the file
mode buys is protection from other local users and from the config
file being pasted into a bug report. `SecretStore` is an interface so
an OS keyring backend can be added later without touching any
provider.
2. **Auto-pick.** Implemented behind `Downloads.AutoPick`, default off.
It requires an MBID-anchored request, match ≥ 0.85, quality ≥ 0.5,
and ≥ 0.08 of daylight over second place. Free-text requests can
never auto-pick, because there is no tracklist to be right about.
3. **Library layout.** Configurable path template, default
`{albumartist}/{album}/{track} {title}`. Segments are sanitized for
Windows-reserved characters and trailing dots/spaces so a library
synced between platforms does not produce unopenable files. Existing
files are never overwritten — a collision gets a numbered variant,
because the file already there may be a better copy the user owns.
4. **Entry point.** "Find this album" on the Explore album page, shown
only when a client is connected and the album is not already owned.
The artist-discography right-click is not wired up yet.
## What shipped
All five phases, ~4,500 lines with tests, `make lint` clean and the full
backend suite green (including under `-race`).
**Core** (`backend/download/`): `Searcher`/`Transporter`/`Delegator`
interfaces with capability-driven composition; `Request`/`Candidate`/
`Result` types; provider registry with self-registering adapters;
two-axis ranking; staging area with escape-guards and startup sweep;
verify → tag → import tail; jobs integration under `KindDownload`;
three tables catalogued in `datamap`.
**Providers**: yt-dlp (subprocess; assembles albums from per-track
searches, since a "full album" video cannot be imported as tracks),
slskd (remote search + transport, peer-health scoring, collects from the
daemon's own downloads folder), Lidarr (delegate; reconciles in place
rather than moving files out from under a system still managing them),
Prowlarr (search-only) paired at grab time with qBittorrent or SABnzbd.
**Frontend**: `download-store.ts`, `download-picker` + `candidate-row`
(two meters, not one blended score), `download-clients` settings section
rendering its forms from backend descriptors so a new adapter needs no
frontend change.
## Follow-ups
- **Resume across restart.** Live transfers are currently marked failed
on startup and their staging swept, because the transports do not
survive the process. slskd and qBittorrent can both resume in
principle; the item rows already carry what would be needed.
- ~~**Per-provider concurrency caps.**~~ Done in 004: per-kind defaults
(slskd 1, yt-dlp 2, torrent/usenet 4) with a per-provider override,
and the provider's slot is taken before the global one.
- **Prowlarr candidates score blind.** Indexer results carry no file
list, so match scoring has only the release title. Fetching the
torrent metadata before ranking would fix this and is the single
biggest ranking improvement available.
- ~~Wanted list, artist monitoring~~ — done in 004. Quality profiles
and upgrade-if-better remain deferred.
.planning/plans/completed/004-wanted-list.md
+163
View File
@@ -0,0 +1,163 @@
# 004 — Wanted list
**Status:** implemented
**Branch:** main
**Created:** 2026-07-29
**Follows:** 003-download-clients
## Problem
Plan 003 shipped a request as a heavyweight row: library, anchors,
cached tracklist, state machine, error text, cascading items. That is
the right shape for *one attempt to acquire something* and the wrong
shape for *the user wanting something*, and 003 used it for both.
The consequences showed up immediately. A request that found nothing was
marked `failed`, which is a lie — the album exists, no source had it
today. Retrying meant the user remembering to press a button. Wanting an
artist's future releases was not expressible at all. And a user who
acquired an album by other means kept a failed row about it forever.
## The model
A **want** is an MBID, what that MBID names, and retry bookkeeping.
That is all.
```
download_wants(mbid, entity, library_id, scope, secondary, state,
parent_id, attempts, last_error, next_try_at,
external_ids)
```
`entity` is the only type distinction, and it carries all the policy:
| entity | meaning |
|---|---|
| `artist` | a subscription. Never satisfied; each pass expands the discography into child wants |
| `release-group` | an album in the abstract — any release satisfies it |
| `release` | one specific edition |
| `recording` | one track |
`UNIQUE(mbid, library_id)` is load-bearing: it is what makes artist
expansion idempotent, so a subscription can re-run every pass and add
only what is genuinely new.
Requests did not go away — they became what they always were, the
ephemeral record of one attempt, with a nullable `want_id` back-link.
The lifetimes are now opposite and explicit: **a request is history, a
want is intent.**
### Nothing here fails
There is no `failed` want state. An attempt can fail; a want cannot. A
want that found nothing gets `attempts + 1`, a reason the user can read,
and a longer backoff — 6h doubling to a 7-day ceiling, jittered so a
list added in one sitting does not come due in one burst.
### Satisfaction is ownership, not download
A want retires when the *library* owns what it names, however it got
there — bought, ripped, copied in. Inferring satisfaction from our own
completed downloads would keep hunting for music already on disk.
### Artist scope defaults to `future`
Following an artist takes new releases only, and skips compilations,
live albums and remixes. `all` backfills the discography, and the user
can widen it from the wanted list. Subscribing should not silently queue
forty albums.
## The reconciler
A 6-hourly loop (plus on-demand, plus a 3-minute startup delay so the
explore index has loaded). Four steps, in this order:
1. **Expand** artist subscriptions into album wants — first, so step 2
sees them this pass rather than next.
2. **Retire** wants the library already owns.
3. **Sync** to clients that keep their own list.
4. **Attempt** a bounded batch (25) of due wants.
Everything the loop needs about music comes through a four-method
`CatalogPort`, adapted to the explore index in `backend/downloadcatalog.go`
— the composition root, so neither package learns about the other.
### Unattended grabs, and what stops them
`Manager.Attempt` is `Start` without the parking: it searches, and grabs
only if `AutoPickable` clears. When it does not, **nothing is
persisted** — no request row. A want retried weekly for a year would
otherwise leave fifty identical failed rows, none of them anything the
user can act on.
`AutoPickable` gained one condition: an anchored request with an empty
`Expected` is refused. An anchor with no tracklist behind it is an
anchor in name only, and match then rests on album/artist text — exactly
the evidence a wrong-album candidate also has. Nobody is watching a
reconcile pass.
## Per-provider concurrency
`Downloads.MaxConcurrent` was the only limit, and was never actually
applied (`SetMaxConcurrent` did not exist). Now:
- **slskd defaults to 1.** A Soulseek peer serves one file at a time
from one person's upload slot; asking for more gets you queued behind
everyone else at best. One is both the polite number and usually the
fastest.
- yt-dlp 2, torrent/usenet clients 4, overridable per provider via a
`maxConcurrent` field that `Register` appends automatically to any
descriptor declaring `CanTransport`.
- A grab takes its **provider's** slot before the global one, so a queue
on a busy slskd cannot sit on a global slot a usenet transfer could
have used. The transport is resolved before either slot is taken;
delegates take neither, since the transfer is happening inside another
system that is doing its own limiting.
## The Lister role
The fourth role, alongside Searcher/Transporter/Delegator. Lidarr
already models a want — a monitored artist or album — and it is always
on, where a desktop player is not. A subscription mirrored there keeps
working while the app is closed.
- `artist` → Lidarr artist, `monitor: future|missing` per scope
- `release-group`/`release` → monitored album
- `recording` → not pushed. Lidarr cannot say "one track", and
monitoring the album to get it downloads far more than was asked.
Sync is push-only in the loop; pulling happens only when the user
explicitly imports ("adopt the artists Lidarr already monitors", which
arrive at `future` scope). Removal **unmonitors**, never deletes — the
user's Lidarr may predate this app.
## Frontend
- `Wanted` view in the sidebar: Following / Looking for / Paused /
Found, with pause, remove, scope toggle and "Check now".
- "Want this" on the album page, "Follow for new releases" on the artist
page. The want button shows whether or not a client is connected —
wanting is durable and stays queued until one exists.
- `WantedListChanged` event, since a background pass changes the list
without the UI doing anything.
## Files
`backend/download/want.go`, `wantstore.go`, `reconcile.go`,
`provider_lidarr_list.go`; `backend/downloadcatalog.go`;
schema `download_wants.sql` + migration 48 for the two new
`download_requests` columns; `frontend/src/components/wanted-view/`.
## Deferred
- **Release-group wants are not retired by ownership of a specific
release.** The library indexes release groups and recordings, not
editions, so a `release` want is only satisfied by its own download
completing.
- **No recording lookup on the explore index**, so a track want relies
on the title the UI passed in. A want added as a bare recording MBID
has no tracklist and waits.
- Quality profiles and upgrade-if-better (from 003).
- Resume across restart (from 003) — still the largest gap, and it now
matters more: an unattended grab that dies on restart is retried by
the reconciler, but from zero bytes.