yellowjacket/.planning/plans/completed/003-download-clients.md

# 003 — Download clients

**Status:** implemented (v1); follow-ups tracked below
**Branch:** main
**Created:** 2026-07-27

## Problem

YellowJacket can find music (`explore`), identify it (`autotag`), and
manage it (`library`) — but it can't acquire it. The one gap between
"you're missing this album" and "you own this album" is filled today by
the user alt-tabbing to some other tool.

The naive fix is an HTTP client for Soulseek and a shell-out to yt-dlp.
That produces two bespoke code paths with duplicated queueing, retry,
staging and import logic, and a third service means a third copy. The
services users want to connect are also not the same *kind* of thing —
some search, some transfer bytes, some are entire automation systems we
delegate to — so a single `DownloadClient` interface would be a lie that
every adapter partially implements.

## The role decomposition

Every candidate integration fills one or two of three roles:

| Service | Searches | Transports | Delegates |
|---|---|---|---|
| slskd (Soulseek) | ✅ | ✅ | |
| yt-dlp | ✅ | ✅ | |
| Lidarr | | | ✅ |
| Prowlarr | ✅ | | |
| qBittorrent / Transmission | | ✅ | |
| SABnzbd / NZBGet | | ✅ | |

So: three small interfaces, not one big one. A provider implements
whichever it supports and declares that in a capability struct, the same
way `jobs.Caps` lets the frontend render controls without switching on
`Kind`.

```go
// Searcher turns a request into ranked candidates.
type Searcher interface {
    Search(ctx context.Context, req Request) ([]Candidate, error)
}

// Transporter moves a candidate's bytes to a local staging directory.
type Transporter interface {
    Grab(ctx context.Context, c Candidate, dst string, p ProgressFunc) (Result, error)
    Cancel(ctx context.Context, grabID string) error
}

// Delegator hands the whole request to an external manager and
// reports back when files land.
type Delegator interface {
    Request(ctx context.Context, req Request) (string, error)
    Poll(ctx context.Context, externalID string) (DelegateStatus, error)
}
```

A `Provider` is the registry entry: identity, config, health check, caps,
plus whichever of the three it satisfies. Search-only providers
(Prowlarr) are paired with a transport at grab time by protocol match
(`torrent` → qBittorrent, `usenet` → SABnzbd); providers that do both
are self-pairing.

## v1 decisions (settled)

- **On-demand only.** User-initiated "find this album" from an Explore
  artist/album page or a missing-album row. No wanted list, no artist
  monitoring, no quality-cutoff upgrades. The queue and pipeline built
  here are exactly what monitoring would later sit on top of — see
  Deferred.
- **Soulseek via slskd's REST API**, not a native protocol client. Same
  adapter shape as everything else, no wire protocol, no credentials in
  our process, fully testable against an `httptest` server. A native
  provider can slot in behind `Searcher`/`Transporter` later with no
  pipeline changes.
- **Stage → autotag → import.** Downloads land in a staging directory,
  are matched against the intended release with the existing `autotag`
  scorer, tagged, then moved into the library and scanned. Never write
  into the library root directly.
- **All four provider families in v1**, sequenced so each phase proves a
  different role shape (see Phases).

## Pipeline

```
Request (MBID-anchored where possible)
  └─> fan-out Search across enabled providers (per-provider timeout)
       └─> merge + rank Candidates
            └─> user picks (or auto-pick above confidence threshold)
                 └─> Grab into staging/<request-id>/
                      └─> verify (audio decodes, expected track count)
                           └─> autotag against the intended release
                                └─> tagwriter writes tags
                                     └─> move into library layout
                                          └─> targeted incremental scan
```

The `Request` should carry a release-group or release MBID whenever the
user started from an Explore page, because that anchor is what makes the
autotag step reliable instead of a second guess. Free-text requests are
supported but flagged lower-confidence, and never auto-pick.

Staging lives under the user data dir, not the library. Partial grabs are
resumable where the provider supports it and swept on startup where it
doesn't.

## Candidate ranking

Two independent scores, kept separate:

1. **Match confidence** — does this candidate contain the release the
   user asked for? Reuse `autotag`'s distance/alignment machinery on the
   candidate's *filenames* (Soulseek gives paths, not tags), against the
   expected tracklist from the explore index.
2. **Source quality** — format (FLAC > V0 > 320 > lower), bitrate,
   completeness (file count vs. expected track count), source health
   (slskd queue length and upload slots; seeders for torrents), and a
   user-set per-provider priority.

Ranking presents both, because they trade off — a perfectly-matched
128kbps rip should lose to a well-matched FLAC, and the user should be
able to see why. Reusing `autotag.ScoreBreakdown`'s "explain the ranking"
pattern here is deliberate.

## Persistence

New tables (migration TBD, next free number):

- `download_providers` — id, kind, name, enabled, priority, config blob
  (JSON), `created_at`. Non-secret config only.
- `download_requests` — id, source (`explore-album`, `explore-artist`,
  `manual`), release_mbid / release_group_mbid, free-text query,
  requested_at, state, resolved_download_id.
- `download_items` — one row per grab attempt: request_id, provider_id,
  candidate JSON, state, bytes/total, staging path, error, timestamps.

**Secrets** (slskd API key, Lidarr/Prowlarr API keys, qBittorrent
password) do not go in the TOML config or the DB in plaintext. Use the OS
keyring where available with a clearly-labelled encrypted-file fallback,
and never log a config value from a provider's secret field. Open
question below on the exact library.

## Jobs integration

Add `jobs.KindDownload`. One job per request (not per file), with
`Stages` for search → grab → import so the existing detail panel renders
the pipeline for free. `Caps{Cancellable: true}`; pausable only for
providers that can resume. Per-provider concurrency caps and a global
cap, both configurable — hammering a Soulseek peer with eight parallel
transfers gets you queued or banned.

## Frontend

- New `download-providers` section in `config-page` (HTMX + templ, same
  as existing settings) for provider CRUD, test-connection, priority.
- New `download-picker` Lit component: the ranked-candidate dialog,
  invoked from Explore album/artist pages and from a missing-album row.
- `download-store.ts` subscribing to the existing `JobsChanged` event —
  no new event channel needed for progress.

## Phases

Each phase is independently shippable and proves a distinct role shape.

1. **Core.** Interfaces, registry, `Request`/`Candidate`/`Result` types,
   staging dir, ranking, the stage→autotag→import tail, jobs wiring,
   schema, secret storage. Ships with a fake provider and full test
   coverage of the pipeline. No real network.
2. **yt-dlp.** Subprocess provider: search + transport, no server for the
   user to run, so it's the fastest path to an end-to-end working
   feature. Proves the local-subprocess shape (binary discovery,
   version checks, stdout progress parsing, sandboxing the arg list).
3. **slskd.** Remote search + transport over REST. Proves the remote
   HTTP shape and is the highest-value source. This is where filename-
   based match confidence earns its keep.
4. **Lidarr.** Delegate. Proves the fire-and-poll shape, where we don't
   own the transfer and the "import" step is really "detect what Lidarr
   already imported and reconcile".
5. **Prowlarr + qBittorrent/SABnzbd.** Proves split search/transport
   pairing — the one case where two providers cooperate on a single
   request.

## Risks and constraints

- **No bundled credentials, no default-on providers, no preconfigured
  indexers.** Every provider is off until the user configures it. The
  app ships the ability to connect to services the user already runs.
- **yt-dlp is a moving target.** Pin a minimum version, check it at
  provider-enable time, and fail with a clear message rather than
  parsing garbage output.
- **Filename-only matching is genuinely hard.** Soulseek results are
  `\Music\Album (1997) [FLAC]\01 - Track.flac` at best. Budget real
  effort for the path-parsing heuristics; `autotag/normalize.go` is the
  starting point.
- **Partial and failed grabs must never reach the library.** The import
  step is the only writer into library paths, and it runs after
  verification. Staging sweep on startup.
- **Tests must not hit the network.** `httptest` servers for slskd/
  Lidarr/Prowlarr, a stub binary for yt-dlp.

## Deferred

- Wanted list with background retry (the natural next plan).
- Artist monitoring + auto-grab of new releases — cheap once the wanted
  list exists, because `explore`'s dump index already knows the full
  discography and `library` already knows what's owned.
- Quality profiles and upgrade-if-better.
- Native Soulseek protocol client.
- Transmission/Deluge/NZBGet (same shape as their shipped siblings —
  add on demand).
- Internet Archive / Bandcamp-collection providers: cheap REST adapters,
  worth adding once the core is proven.

## Resolved questions

1. **Secret storage.** No keyring dependency was added. Credentials go
   in a 0600 JSON file in the user data directory (`download-secrets.json`),
   keyed by provider row ID. This is deliberately *not* encryption — a
   key stored beside the data it unlocks protects nothing, and claiming
   otherwise would be worse than being clear about it. What the file
   mode buys is protection from other local users and from the config
   file being pasted into a bug report. `SecretStore` is an interface so
   an OS keyring backend can be added later without touching any
   provider.
2. **Auto-pick.** Implemented behind `Downloads.AutoPick`, default off.
   It requires an MBID-anchored request, match ≥ 0.85, quality ≥ 0.5,
   and ≥ 0.08 of daylight over second place. Free-text requests can
   never auto-pick, because there is no tracklist to be right about.
3. **Library layout.** Configurable path template, default
   `{albumartist}/{album}/{track} {title}`. Segments are sanitized for
   Windows-reserved characters and trailing dots/spaces so a library
   synced between platforms does not produce unopenable files. Existing
   files are never overwritten — a collision gets a numbered variant,
   because the file already there may be a better copy the user owns.
4. **Entry point.** "Find this album" on the Explore album page, shown
   only when a client is connected and the album is not already owned.
   The artist-discography right-click is not wired up yet.

## What shipped

All five phases, ~4,500 lines with tests, `make lint` clean and the full
backend suite green (including under `-race`).

**Core** (`backend/download/`): `Searcher`/`Transporter`/`Delegator`
interfaces with capability-driven composition; `Request`/`Candidate`/
`Result` types; provider registry with self-registering adapters;
two-axis ranking; staging area with escape-guards and startup sweep;
verify → tag → import tail; jobs integration under `KindDownload`;
three tables catalogued in `datamap`.

**Providers**: yt-dlp (subprocess; assembles albums from per-track
searches, since a "full album" video cannot be imported as tracks),
slskd (remote search + transport, peer-health scoring, collects from the
daemon's own downloads folder), Lidarr (delegate; reconciles in place
rather than moving files out from under a system still managing them),
Prowlarr (search-only) paired at grab time with qBittorrent or SABnzbd.

**Frontend**: `download-store.ts`, `download-picker` + `candidate-row`
(two meters, not one blended score), `download-clients` settings section
rendering its forms from backend descriptors so a new adapter needs no
frontend change.

## Follow-ups

- **Resume across restart.** Live transfers are currently marked failed
  on startup and their staging swept, because the transports do not
  survive the process. slskd and qBittorrent can both resume in
  principle; the item rows already carry what would be needed.
- ~~**Per-provider concurrency caps.**~~ Done in 004: per-kind defaults
  (slskd 1, yt-dlp 2, torrent/usenet 4) with a per-provider override,
  and the provider's slot is taken before the global one.
- **Prowlarr candidates score blind.** Indexer results carry no file
  list, so match scoring has only the release title. Fetching the
  torrent metadata before ranking would fix this and is the single
  biggest ranking improvement available.
- ~~Wanted list, artist monitoring~~ — done in 004. Quality profiles
  and upgrade-if-better remain deferred.