Compare commits

..
Author SHA1 Message Date
yonluandClaude Opus 5.5 399dcc05b3 docs(notes): record slskd's API as read from its source
CI / check (push) Skipped
CI / e2e (push) Skipped
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017HJiuc3ZZhxsPXz3ozTirT
2026-09-26 22:11:30 -04:00
yonluandClaude Opus 5.5 ef89707bb1 feat(download): fill in the tracks a nearly complete album is missing
A grab that delivers nine of twelve tracks clears the completeness floor
and is imported, and the other three were never looked for. On Soulseek
that is the commonest way an album ends up almost right: one peer's
folder lacks a track, or one file fails.

After a successful import the manager now compares the tracks the files
were aligned to (ImportResult.Matched, new) with the expected tracklist.
When one to three are missing, and fewer than half, it searches for each
one on its own, as the track's artist and title with the album kept for
ranking. It grabs the first auto-acceptable copy that is not from the
source that already failed to supply it, and is not a delegate, which
would place it in its own library. The candidate is trimmed to the one
file aligned to the track. The import uses the album's own request with
ImportOptions.Only, which skips the completeness check and imports only
a file aligned to the missing track, so it is tagged and placed as part
of the album, and anything else the folder brought is left out.

One attempt per track, and nothing here fails the download: the album
is already imported, so a track that cannot be found is logged on the
job and left. slskd accepts one-file folders for a request that expects
one track, which the per-track search needs.

Closes #276

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017HJiuc3ZZhxsPXz3ozTirT
2026-09-26 22:11:03 -04:00
yonluandClaude Opus 5.5 c88fc1c7c7 feat(download): judge a queued slskd peer by its queue position
No bytes for ten minutes usually means a peer has queued us, and the
stall timer could not tell position 2 from position 400: the first was
abandoned while it was about to start, the second was waited on for ten
minutes for nothing.

While a requested file is "Queued, Remotely", the grab asks slskd for
its place (GET .../downloads/{user}/{id}/position, which asks the peer)
once a minute. A place that improved counts as progress and restarts
the stall clock. A place beyond 50 twice running gives the peer up at
once, and the manager moves to the next copy; two readings because
slskd documents the figure as possibly inaccurate. Waiting in a queue
has an overall ceiling of an hour without a byte, since a queue moving
one place an hour would otherwise hold the grab all day. As with a
stall, files that already arrived still go forward.

Closes #275

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017HJiuc3ZZhxsPXz3ozTirT
2026-09-26 22:07:18 -04:00
yonluandClaude Opus 5.5 7a9dd69d30 fix(download): own folder per slskd grab, search timeout in seconds
Checked against slskd 0.26.0's source rather than a live daemon, which
#267 never had.

slskd reads a search's searchTimeout in seconds, counted from the last
response. We sent milliseconds, telling it a search may idle for five
hours, so a search never completed on its own. It now sends seconds,
with slskd's floor of 5.

slskd writes a finished file to <downloads>/<remote leaf folder>/, and
when a name is taken it writes name_<ticks>.ext beside it. collect found
files by name there, so a file left by an earlier failed attempt, or by
the user's own download, was collected in place of this grab's. A user
who changed slskd's destination setting got nothing collected at all.

slskd 0.26 takes a batch download with an explicit destination. Each
grab now enqueues batches into yellowjacket/<uuid>/ (one per disc, since
a batch's files land flat), collects from exactly there, and removes the
folder afterwards, including after a failure. An older daemon answers
the batch route with 400, which is remembered, and the per-user enqueue
is used. There, collect skips files that were already present,
unchanged, before the enqueue, and takes the renamed copy slskd wrote
instead. The comparison is against a snapshot, not a clock, because
slskd may run on another machine. The per-folder lock from #272 is kept
only while batches are not known to work.

The test stub now writes files when they are enqueued, as slskd does,
including the rename, so tests no longer stage files before a grab.
An opt-in TestSlskdLive runs against a real daemon when YJ_SLSKD_URL,
YJ_SLSKD_API_KEY and YJ_SLSKD_DOWNLOADS are set.

Closes #274

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017HJiuc3ZZhxsPXz3ozTirT
2026-09-26 22:04:50 -04:00
yonlu 3f23bb4396 Merge pull request 'Batch: explore catalog + UI, Go 1.26, Soulseek downloads, CI fixes' (#273) from batch/258-272 into main
CI / check (push) Successful in 4m8s
CI / e2e (push) Successful in 14m37s
Reviewed-on: #273
2026-09-27 01:45:18 +00:00
yonlu af2ff17342 Merge remote-tracking branch 'origin/fix/263-slskd-transfer-lifecycle' into batch/258-272
CI / check (push) Skipped
CI / e2e (push) Skipped
CI / check (pull_request) Successful in 4m53s
CI / e2e (pull_request) Successful in 13m34s
2026-09-26 20:58:54 -04:00
yonlu fc19ca54b7 Merge remote-tracking branch 'origin/build/265-go-1.26' into batch/258-272 2026-09-26 20:58:54 -04:00
yonlu 613901847a Merge remote-tracking branch 'origin/feat/264-explore-cards' into batch/258-272 2026-09-26 20:58:54 -04:00
yonlu ecf0109331 Merge remote-tracking branch 'origin/fix/258-artifact-blob-cursor' into batch/258-272 2026-09-26 20:58:53 -04:00
yonluandClaude Opus 5.5 9710c11476 feat(download): one grab per Soulseek peer, several peers at once
CI / check (push) Skipped
CI / e2e (push) Skipped
CI / check (pull_request) Failing after 2m51s
CI / e2e (pull_request) Skipped
slskd was capped at one transfer per daemon, on the grounds that
Soulseek peers punish clients that ask for too much. That politeness is
per peer: two different users do not compete for anyone's upload slot.
So one slow peer serialised every other Soulseek download behind it.

The manager now takes a per-(provider, peer) lock before any slot, so a
grab waiting on a busy peer does not hold a provider slot another peer
could use, and the slskd default rises to 3, which now counts peers.
The help text says so.

Running grabs at once exposed the folder collision: slskd names a
download's directory after the remote leaf folder, so two peers'
"Greatest Hits" (or any two rips' "CD1") share one directory, and
collect finds files by name there. Grabs whose local folders overlap
now take a package-level lock per folder, in sorted order, keyed on the
full path because two clients can share one daemon.

Closes #272

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017HJiuc3ZZhxsPXz3ozTirT
2026-09-26 17:31:35 -04:00
yonluandClaude Opus 5.5 5e3ac8fb1b fix(download): search Soulseek more than once, and read file lengths
CI / check (push) Skipped
CI / e2e (push) Skipped
CI / check (pull_request) Canceled after 0s
CI / e2e (pull_request) Canceled after 0s
The slskd search asked one question and ignored part of the answer.

Two queries.  Soulseek matches every term against a file's full path,
so every extra word is a filter, and several filter wrongly: an edition
qualifier from the catalog title that no one puts in a folder name, a
term with a leading "-", which Soulseek reads as an exclusion, and
"Various Artists", which is in no one's path.  When a normalised form of
the request differs, it runs alongside the original and the candidates
are merged by peer and folder.  Concurrently, not as a fallback: the
manager gives a provider one search budget, and a Soulseek search spends
most of it waiting.  A query the user typed is searched as written.

Stated options.  The search carried only its id and text, so slskd's
own defaults for its timeout and response limits applied.  Its timeout
is now set inside our wait, the limits are well above a popular album,
and slskd drops folders below the file floor and peers with a queue we
would not reach today.

A state-only poll.  Every one-second poll re-sent every response; the
responses are now fetched once at the end, falling back to the old
includeResponses form for a daemon without that endpoint.

Durations.  slskd reports each file's length and it was discarded.  It
is now carried as CandidateFile.LengthMillis and scored against the
expected tracks as DurationFit, which takes 0.15 of title fit's weight
when at least half the aligned pairs are timed: a title says which song
a file claims to be, a length says whether it is that recording.
Without lengths the score is exactly the previous formula.

freeUploadSlots is removed from the response type; slskd sends
hasFreeUploadSlot and nothing by that name.

Closes #271

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017HJiuc3ZZhxsPXz3ozTirT
2026-09-26 17:11:23 -04:00
yonluandClaude Opus 5.5 f81a950916 fix(download): score a multi-disc rip as one album, and count tracks
CI / check (push) Skipped
CI / e2e (push) Skipped
Three faults in how candidates are shaped and scored, one commit because
they meet in the same completeness number.

Multi-disc albums were split in two.  Soulseek shares them as
Album/CD1 and Album/CD2, and candidates were grouped by the immediate
parent, so each disc became its own candidate titled "CD1": about half
complete, with an album title that could not match.  Such a release
essentially never cleared auto-pick.  AlbumDir groups a disc folder
under its parent, ParsePath takes the disc number from the folder (a
disc in the filename still wins), and collect keeps the disc folders in
staging, where flattened, disc 2's "01 Intro.flac" overwrote disc 1's.

A single-track request could never be served from Soulseek.  A track
search matches one file per folder, and the two-file floor that screens
out noise for an album screened out every result.  A recording request
takes one.

Completeness counted files.  Ten files against a ten-track album scored
full marks whether or not they were its tracks, and title fit is the
mean over the files that did align, so a folder where three titles
matched read as near-perfect on both.  Coverage is now counted in
aligned tracks, with the file count still setting the penalty for
extras.

Closes #270

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017HJiuc3ZZhxsPXz3ozTirT
2026-09-26 17:07:28 -04:00
yonluandClaude Opus 5.5 7fbfd9c105 test(library): skip the cover-tier scan test when fixtures are absent
CI / check (push) Skipped
CI / e2e (push) Skipped
CI / check (pull_request) Successful in 4m23s
CI / e2e (pull_request) Canceled after 0s
TestScan_StoresOnlyCoverTiers built the fixture path by hand, so in a
tree where make testdata had not run it failed on a missing covers
directory, where every other fixture test skips via testfixtures.Load.
In a fresh worktree that failure blocked the pre-push hook for every
branch.

Closes #266

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017HJiuc3ZZhxsPXz3ozTirT
2026-09-26 17:02:56 -04:00
yonluandClaude Opus 5.5 792c2d9fbc build: require Go 1.26 everywhere at once
CI / check (push) Skipped
CI / e2e (push) Skipped
CI / check (pull_request) Successful in 8m21s
CI / e2e (pull_request) Successful in 15m56s
The newest go-json-experiment/json, which wails/v3's application
package imports, declares go 1.26, so taking it raises our go
directive with it.  Every other place that names a Go version moves in
the same commit: GO_VERSION in ci, desktop-assets and android-apk, the
Arch makedepends, and CONTRIBUTING's table.

index-artifact's container image moves too, and it is the one that
matters most.  The official golang images set GOTOOLCHAIN=local, so a
golang:1.25 container refuses a go 1.26 module outright rather than
fetching a newer toolchain, and that job owns the ~205 GB checkpoint.

Closes #265

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017HJiuc3ZZhxsPXz3ozTirT
2026-09-26 17:00:47 -04:00
yonluandClaude Opus 5.5 0a33b9d653 fix(download): try the next acceptable copy when a transfer fails
A failed transfer failed the whole download.  On Soulseek the usual
failure is one peer being offline or refusing, and a popular album has
several other peers offering the same folder; the ranked list that
names them was already held in m.results and nothing walked it.

grab now loops: when a candidate's transfer fails, or delivers too
little of the album to import, the next candidate is tried in its
place, up to three in all.  Three rules keep that honest:

- Only a candidate auto-pick would itself have accepted is offered, so
  a second choice clears the same match, quality and guardrail gates
  as the first.
- On Soulseek the failure is the peer's, so every folder that peer
  offered is skipped with it; elsewhere only the failed release is.
- A candidate the user picked by hand does not fall back.  They chose
  that copy, and quietly substituting another is a decision they did
  not make.

The same change fixes auto-pick grabbing the wrong candidate.
AutoPickVeto judges the best candidate inside the user's guardrails,
but Start and Attempt then grabbed ranked[0] -- so when the overall best
was over the size ceiling, the veto passed on the strength of the
second and the first was downloaded anyway: the one copy the user had
said not to take unattended.  autoPick returns the candidate the veto
actually judged.

Closes #263

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017HJiuc3ZZhxsPXz3ozTirT
2026-09-26 16:12:06 -04:00
yonluandClaude Opus 5.5 fc0121228e fix(download): give up on a stalled slskd peer and cancel its transfers
awaitTransfers waited on every requested file reaching a terminal state
with no bound but the caller's six-hour context.  slskd's transfer
limit is one, so a peer that queued us and never sent a byte held every
other Soulseek download behind it for the whole six hours.  A grab now
gives up after ten minutes with no bytes moving; a folder that stalls
on its last tracks goes forward with what arrived, as a partial failure
always has.

Three smaller faults on the same path:

- A file slskd never lists (refused at enqueue) could never reach a
  terminal state, so the wait could not end.  It counts as failed after
  a short grace period.
- A terminal record left by an earlier attempt at the same file from the
  same peer was read as this attempt's answer on the first poll.  The
  ids already terminal before enqueue are ignored.
- Giving up, for any reason, left slskd downloading for a request nobody
  was waiting on.  The live transfers are cancelled and removed there,
  on a context of their own so a cancelled caller still sends it.

Usernames are now path-escaped; they may carry spaces and slashes.

Refs #263

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017HJiuc3ZZhxsPXz3ozTirT
2026-09-26 16:11:52 -04:00
yonlu 924097b246 Merge branch 'docs/256-claude-md-split' into fix/258-artifact-blob-cursor
CI / check (push) Skipped
CI / e2e (push) Skipped
CI / check (pull_request) Successful in 3m57s
CI / e2e (pull_request) Successful in 14m49s
Both branches rewrite CLAUDE.md. Mine appended a note about #258 to the
"the catalog stores ids as bytes" section; theirs cuts the file from
4,046 lines to 376 and deletes those write-ups, on the grounds that each
one already exists as a comment beside the code it describes.

Resolved in theirs. Nothing is lost by it: the refactor already states
both halves of the note as broad engineering rules, and cites #258 —
"Encodings are converted at one boundary … a Go `string` cursor against
a BLOB column made the catalog merge loop forever while merging nothing"
and "A loop that must make progress checks that it did … a merge that
lands fewer rows than it read". The detail stays in `artifactKey`'s doc
comment and in `publishedartifact_test.go`'s, which is where the new
file says this class belongs.
2026-09-26 15:56:44 -04:00
yonluandClaude Opus 5.5 1997276def docs: cut CLAUDE.md to the rules it is for
CI / check (push) Skipped
CI / e2e (push) Skipped
CI / check (pull_request) Successful in 4m22s
CI / e2e (pull_request) Successful in 14m24s
CLAUDE.md had grown to 4,046 lines, ~3,300 of them per-component
write-ups: why a breakpoint is 500px, why a cap is a quarter, what a
spec once missed. Every session loaded all of it, and the rules that
apply to every change were buried among decisions that apply to one.

Those write-ups were already duplicated as comments beside the code
they describe -- every issue number they cite also appears in a code
comment -- so they are deleted rather than moved. What is left is the
tracker workflow, the commands, the verification tiers and a set of
broad engineering rules distilled from them, each pointing at where
its example lives. Three declined decisions recorded nowhere else go
to NOTES.md, and three code comments that cited CLAUDE.md sections by
name no longer do.

Closes #256

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-26 15:47:13 -04:00
yonlu 5d9c677cf7 ci(index-artifact): import the exported artifact before publishing it
CI / check (push) Skipped
CI / e2e (push) Skipped
CI / check (pull_request) Successful in 3m59s
CI / e2e (pull_request) Successful in 12m51s
Nothing should be published until the code that imports it on a user's
machine has imported it here. The exporter and the importer are two
descriptions of one storage format, and every other tier tests the
importer against a *fixture* rather than against the file being shipped —
a second description free to be wrong in the same direction as the code
reading it.

That is how #258 reached everyone: the importer positioned its batch walk
with a Go `string` cursor against this file's 16-byte `mbid` column, and
SQLite neither coerces between TEXT and BLOB nor complains about the
comparison, so the walk merged no rows and never advanced. The fixture
guarding that walk writes the old text encoding, and the only compact
fixture is one row, below the batch size, so the bound query never ran.
Both were green throughout, and no install could finish a first index
build.

`TestImportPublishedArtifact` takes the published file and runs the
client's own path over it — checksum, decompress, merge — and asserts
that what the artifact holds is what the client ends up with: the row
count, the rows carrying a listen count, an FTS index in step with the
table, and one row read back through the app's own MBID conversions. It
skips without `YJ_CORE_INDEX_ARTIFACT`, so an ordinary run pays nothing.

It needs no Wails, which is why the step runs it under the indexbuild
tag: that container has no GTK. Measured on the current artifact, 64.8 MB
compressed: 39 seconds including the decompress.

Verified against the pre-fix comparison behaviour, the step goes red in
about 3 seconds — the strictly-advancing guard fails the import with a
named reason rather than the day-long spin it used to produce.

Refs #258
2026-09-25 11:03:52 -04:00
yonlu 1e3a490c12 fix(explore): refuse a catalog merge that does not land every row
The walk's predicates partition the artifact's key space, so a merge that
ends with fewer rows than the artifact declares does not mean the artifact
was smaller than it said — it means a predicate filtered rows out, and
the catalog is quietly partial while reporting complete.

Equality rather than a lower bound: RowsAffected counts an upsert that
changes nothing, and a row already merged locally is counted again here.

One reachable case, so this is not merely a tripwire. A row whose mbid is
empty is excluded by `mbid > ?` in both encodings, so an artifact
carrying one imports as a success with a row missing — which is the shape
#258 had, one cause over. The test covers exactly that artifact.

Refs #258
2026-09-25 11:03:33 -04:00
yonlu d4ea14ca5c fix(explore): merge the catalog artifact in its own mbid encoding
CI / check (push) Skipped
CI / e2e (push) Skipped
CI / check (pull_request) Successful in 4m4s
CI / e2e (pull_request) Successful in 14m14s
The prebuilt catalog never merged. `mergeArtifactRows` positions itself
with `WHERE mbid > ? ORDER BY mbid LIMIT 1 OFFSET ?` against the
attached artifact, and it bound that cursor as a Go `string` while
`cmd/indexexport` publishes `explore_index.mbid` as 16 raw bytes — the
storage change that took the table from 677 MB to 389 MB.

SQLite does not coerce between TEXT and BLOB and orders every blob after
every text value, so against a byte column the predicate was not wrong
but unconditional: `mbid > <text>` matched the whole artifact, so the
bound the walk looked up was the same row every time and the cursor
never advanced, and `mbid <= <text>` matched nothing, so no batch
merged. No error, no rows, no state change — a fresh install sat at "0
of 1,077,893 rows" burning a core indefinitely, which is what it did
here for a day, while Explore showed only the rows the library scan and
the lazy artist enrichment had produced and popularity for none of the
catalog.

The cursor is now an `artifactKey`, typed to the encoding
`artifactStoresText` reports for the file it is attached to, so the
comparison is made in the same type as the column it is made against.
Two things guard the class rather than the instance: a nil key binds as
an empty value instead of SQL NULL, because `mbid > NULL` agrees with
nothing and would import nothing just as silently; and the walk returns
an error when its bound does not strictly advance, because the failure
here is silence and the next one should be a failed job with a reason.

It was never caught because the fixture that guards the walk writes the
old text encoding, and the only compact fixture is a single row — below
`artifactMergeBatch`, so the bound query never ran at all. The walk is
now covered on both encodings, across several batch boundaries.

Closes #258
2026-09-25 10:18:51 -04:00
yonlu 62c1a95ead test(e2e): give the job specs their own state back
Both specs stage a job through `/__test/emit` and neither cleared it.
Nothing resets those stores, so the spec that staged it is the one that
should put it back, and `JobsChanged` with `[]` is the whole cleanup --
`JobStore` replaces its list from every snapshot, so `testctl` needs no
special case.

**The leak as reported did not reproduce, and that is worth recording
rather than quietly fixing.**  Measured with a temporary probe: a
positive control confirmed a staged job really does move the shell at a
phone width (`job-band` renders a row, `.main-panel`'s top goes 0 to
55), and the very next page had no job at all.  The reason is that
every test gets a fresh page and `JobStore.init()` refetches `GetJobs()`
from a backend registry `/__test/emit` never writes to -- it calls
`events.Deliver`, which touches frontends and no state.  So the state
cannot cross a spec boundary as described, and the 55px offset the
draft assertion saw in that suite run has another cause that is not in
evidence.

The cleanup stays, because it costs a line and the leak would need only
one spec that keeps a page alive, and the comments say what was
measured rather than asserting the mechanism.

The durable half is the rule, now in the harness reference: measure
against the element next to you, not an absolute coordinate.  An
absolute number in a shell measurement is also a claim about everything
above it -- `contentTop === 0` asserts "and no background job is
running", which that spec could not arrange.

Closes #168
2026-09-23 07:51:07 -04:00
yonlu e67462ab53 ci: lint every commit a PR would merge, not just its tip
Gitea leaves `github.event.before` empty on a `pull_request`, so the
Commit messages step fell through to bare `make commit-check`, which
lints `git log -1` -- the tip alone.  Every other commit the branch
would bring was first examined by *main's* post-merge run, so a green
PR stopped being true after the merge, and it happened twice: PR #245
merged a 75-char subject its own CI never saw.

The PR's base is the stand-in.  `base.sha..head` lints the PR's own
commits because base advances on main, so the commits the branch shares
with it stay reachable from it and drop out of the range.

Both payload fields are handed to the shell rather than chosen in an
expression: `github.event.issue.number` in unclaim.yml is this repo's
proof that payload fields resolve, and `github.event` is the webhook
body unmarshalled into a map, so `pull_request.base.sha` comes from
Gitea's own `PRBranchInfo.Sha`.  The shell then falls back to today's
behaviour for a dispatch run, an all-zeros push, or a base commit the
clone does not have -- so the worst case is the fix not taking effect
rather than a broken job.

Verified locally against the report's own evidence: at 68e7edb8 the old
invocation passes ("HEAD is well-formed") while the range catches
a3b5b437 at 75 chars, which is what main's post-merge run did.  All
four event shapes were exercised against the new snippet.  The
end-to-end proof is the next PR with an over-length commit that is not
its tip.

Closes #254
2026-09-23 07:50:59 -04:00
yonlu e5dc54d0ec ci(skill-check): find a make target inside a hard-wrapped span
`scripts/skill-check.sh` matched one regex against one line, so a
mention the file hard-wraps -- `` `make `` at the end of one line and
the target at the start of the next -- was invisible to it.  These docs
are mostly hard-wrapped prose, so the wrap is what the author does not
think about, and `CONTRIBUTING.md:80` is already that shape.

Lines are now joined while the inline span is still open, which an odd
number of backticks means.  The fence and line-start halves are
untouched: a fenced command is already whole, and joining inside one
would break the rule that made this awk rather than a grep.  Joining is
bounded three ways -- a fence, a blank line (CommonMark allows no blank
line inside a code span) and a file boundary -- so a stray backtick
costs one paragraph of over-matching rather than the rest of the file.

The reporting loop needed the other half of the same fix: it named the
offending file with `grep -ln "make $t"`, which cannot see a wrapped
mention either, so a target the new parser found reported no file at
all and `set -o pipefail` turned the empty grep into exit 123 before
the line telling the author what to do.  It falls back to the bare
name.

Verified by planting the report's own wrapped `make
no-such-wrapped-target` into `CONTRIBUTING.md`: the old script reports
48 targets and exits 0, the new one names the target and the file and
exits 1.  Plant removed afterwards.

Closes #228
2026-09-23 07:50:51 -04:00
43 changed files with 5355 additions and 4361 deletions
+1 -1
View File
@@ -68,7 +68,7 @@ jobs:
SHA: ${{ github.sha }} SHA: ${{ github.sha }}
REF_NAME: ${{ github.ref_name }} REF_NAME: ${{ github.ref_name }}
DEBIAN_FRONTEND: noninteractive DEBIAN_FRONTEND: noninteractive
GO_VERSION: '1.25.0' GO_VERSION: '1.26.0'
npm_config_store_dir: /cache/pnpm-store npm_config_store_dir: /cache/pnpm-store
# The Go half wants the NDK; the Gradle half wants a platform. # The Go half wants the NDK; the Gradle half wants a platform.
ANDROID_HOME: /cache/android-sdk ANDROID_HOME: /cache/android-sdk
+23 -6
View File
@@ -36,7 +36,7 @@ concurrency:
cancel-in-progress: true cancel-in-progress: true
env: env:
GO_VERSION: '1.25.0' GO_VERSION: '1.26.0'
# Shared by all three Playwright consumers (@playwright/cli, e2e/'s # Shared by all three Playwright consumers (@playwright/cli, e2e/'s
# @playwright/test, frontend/'s Vitest provider). See the browsers # @playwright/test, frontend/'s Vitest provider). See the browsers
# step in job 2 for why that is not the whole story. # step in job 2 for why that is not the whole story.
@@ -53,7 +53,7 @@ jobs:
check: check:
runs-on: ubuntu-latest runs-on: ubuntu-latest
container: container:
# Not golang:1.25 — this job runs `make ui-test`, which is Vitest # Not golang:1.26 — this job runs `make ui-test`, which is Vitest
# *browser* mode and needs a Chromium and its system libraries # *browser* mode and needs a Chromium and its system libraries
# anyway, so the "fast job needs no browser" split does not hold. # anyway, so the "fast job needs no browser" split does not hold.
# Not the Playwright image either: e2e/ pins @playwright/test # Not the Playwright image either: e2e/ pins @playwright/test
@@ -107,15 +107,32 @@ jobs:
# Conventional Commits. `.releaserc.yml` has always derived the # Conventional Commits. `.releaserc.yml` has always derived the
# version from the commit type; until now nothing checked that the # version from the commit type; until now nothing checked that the
# type was one it recognises, so a malformed subject silently meant # type was one it recognises, so a malformed subject silently meant
# "no release". BEFORE is the push's previous tip and is absent or # "no release".
# all-zeros for a new branch, in which case only the tip is linted. #
# **On a `pull_request` there is no `before`.** Gitea leaves
# `github.event.before` empty for one, so this step fell through to
# bare `make commit-check`, which lints `git log -1` — the tip
# alone. Every other commit the branch would bring was first
# examined by *main's* post-merge run, which is a green PR that
# stops being true after the merge, and which happened twice (#254).
# The PR's base is the stand-in: the range below already excludes
# what the base shares with the branch, because base advances on
# main and those commits stay reachable from it.
#
# Both are handed to the shell rather than chosen in an expression:
# `github.event.issue.number` in unclaim.yml is this repo's proof
# that payload fields resolve, and the shell then falls back to
# today's behaviour for a dispatch run or a missing field instead of
# depending on how `&&`/`||` treat an absent context.
- name: Commit messages - name: Commit messages
working-directory: /src working-directory: /src
env: env:
BEFORE: ${{ github.event.before }} PR_BASE: ${{ github.event.pull_request.base.sha }}
PUSH_BEFORE: ${{ github.event.before }}
run: | run: |
set -eu set -eu
if [ -n "${BEFORE:-}" ] && [ "${BEFORE#0000000}" = "$BEFORE" ] \ BEFORE="${PR_BASE:-${PUSH_BEFORE:-}}"
if [ -n "$BEFORE" ] && [ "${BEFORE#0000000}" = "$BEFORE" ] \
&& git cat-file -e "$BEFORE^{commit}" 2>/dev/null; then && git cat-file -e "$BEFORE^{commit}" 2>/dev/null; then
make commit-check RANGE="$BEFORE..$SHA" make commit-check RANGE="$BEFORE..$SHA"
else else
+1 -1
View File
@@ -48,7 +48,7 @@ jobs:
SHA: ${{ github.sha }} SHA: ${{ github.sha }}
REF_NAME: ${{ github.ref_name }} REF_NAME: ${{ github.ref_name }}
DEBIAN_FRONTEND: noninteractive DEBIAN_FRONTEND: noninteractive
GO_VERSION: '1.25.0' GO_VERSION: '1.26.0'
npm_config_store_dir: /cache/pnpm-store npm_config_store_dir: /cache/pnpm-store
steps: steps:
# The same set ci.yml's check job installs: the app is cgo, and # The same set ci.yml's check job installs: the app is cgo, and
+44 -1
View File
@@ -68,7 +68,7 @@ jobs:
# claim with a test behind it now (cmd/indexbuild/deps_test.go), # claim with a test behind it now (cmd/indexbuild/deps_test.go),
# because the v3 migration quietly broke it and this job was where # because the v3 migration quietly broke it and this job was where
# that surfaced. # that surfaced.
image: golang:1.25 image: golang:1.26
# This host path must exist on the runner and be listed verbatim in # This host path must exist on the runner and be listed verbatim in
# act_runner's container.valid_volumes. It holds explore-staging/ # act_runner's container.valid_volumes. It holds explore-staging/
# (counts.bin + state.json) and yj.db — the checkpoint that makes # (counts.bin + state.json) and yj.db — the checkpoint that makes
@@ -148,6 +148,49 @@ jobs:
sha256sum /tmp/core-index.db.zst | tee /tmp/core-index.db.zst.sha256 sha256sum /tmp/core-index.db.zst | tee /tmp/core-index.db.zst.sha256
ls -lh /tmp/core-index.db.zst ls -lh /tmp/core-index.db.zst
# Nothing is published until it has been imported by the code that
# imports it on a user's machine. The exporter and the importer are
# two descriptions of one storage format, and every other tier tests
# the importer against a *fixture* rather than against the file being
# shipped — a second description free to be wrong in the same
# direction as the code reading it.
#
# That is how #258 reached everyone: the importer positioned its batch
# walk with a Go `string` cursor against this file's 16-byte `mbid`
# column, and SQLite neither coerces between TEXT and BLOB nor
# complains about the comparison — so the walk merged no rows and
# never advanced, and no install could finish its first index build.
# The fixture guarding that walk writes the old text encoding, and the
# only compact fixture is one row, below the batch size, so the bound
# query never ran. Both were green throughout.
#
# Running it here is also what keeps the failure cheap: the previous
# artifact stays published while this runs, so a failure costs one
# stale catalog rather than an empty one for every install.
#
# `-tags indexbuild` because this container has no GTK and the default
# tag set links the app through Wails. The `--- PASS` grep is not
# decoration — the test skips without the path, and a skip is
# indistinguishable from a pass in a summary line.
- name: Import the exported artifact as a client does
if: steps.maintain.outputs.complete == 'true' && steps.maintain.outputs.changed == 'true'
working-directory: /src
env:
YJ_CORE_INDEX_ARTIFACT: /tmp/core-index.db.zst
run: |
set -eu
log=/tmp/import-check.log
if ! go test -tags indexbuild -count=1 -timeout 30m -v \
-run TestImportPublishedArtifact ./backend/explore/ > "$log" 2>&1;
then
tail -60 "$log"
echo "::error::The artifact does not import; not publishing it."
exit 1
fi
cat "$log"
grep -qF -- 'PASS: TestImportPublishedArtifact' "$log"
echo "::notice::The artifact imports as a client would merge it."
- name: Publish to the Gitea package registry - name: Publish to the Gitea package registry
if: steps.maintain.outputs.complete == 'true' && steps.maintain.outputs.changed == 'true' if: steps.maintain.outputs.complete == 'true' && steps.maintain.outputs.changed == 'true'
run: | run: |
@@ -57,6 +57,21 @@ behind `YJ_TESTCTL=1`, which `scripts/dev-headless.sh` sets and
staging the work that would produce it — job progress, download staging the work that would produce it — job progress, download
progress, scan progress. It calls `events.Deliver`, which *errors* progress, scan progress. It calls `events.Deliver`, which *errors*
when the event reaches nobody, so a `200` means it really arrived. when the event reaches nobody, so a `200` means it really arrived.
- **State you stage, you own** (#168). Nothing resets those stores, so
clear yours in `test.afterEach` with the same event that staged it
(`emit('JobsChanged', [])`) — the store replaces its list from every
snapshot, so `testctl` needs no special case. **Measured: this does
not currently cross a spec boundary**, because every test gets a fresh
page and `JobStore.init()` refetches `GetJobs()` from a backend
registry that `/__test/emit` never writes to. Stated anyway, because
it costs one line and the leak needs only one spec that keeps a page
alive — but do not cite #168 for a symptom you have not reproduced.
- **Measure against the thing next to you, not an absolute
coordinate.** An absolute number in a shell measurement is also a
claim about everything above it — `contentTop === 0` quietly asserts
"and no background job is running", which is not what that spec was
about or could arrange, while `contentTop === jobBandBottom` is true
either way. This is the half of #168 that stands on its own.
- **`restore` is slow** (~40 s in the suite) because it copies every - **`restore` is slow** (~40 s in the suite) because it copies every
table. Prefer snapshotting once and restoring only when a spec table. Prefer snapshotting once and restoring only when a spec
genuinely mutates state. genuinely mutates state.
+52
View File
@@ -5078,3 +5078,55 @@ last rendered card.
("ask the virtualizer for a larger overscan") is therefore not ("ask the virtualizer for a larger overscan") is therefore not
available without patching a private, which is why the request is available without patching a private, which is why the request is
issued ahead of the element instead. issued ahead of the element instead.
## Declined, and recorded nowhere else (2026-09-25, #256)
When CLAUDE.md was cut down to rules, most of its "considered and
declined" paragraphs already had a home in a code or config comment
beside what they explain. These three did not:
- **`touch-action: manipulation` was declined** (#54). The 300ms tap
delay it is offered for is already absent on a `width=device-width`
viewport; what it would actually change is the gesture stack #63 tuned
by measurement on the reference device.
- **There is no "Go to Genre"** in the phone row menu (#67). That menu
replaces name links a phone cannot use, and there has never been a
genre link to replace — it would be new navigation, which wants its
own issue.
- **The overlaid queue has no tap-outside gutter on a phone** (#171).
The drawer-style gutter would buy the affordance by taking width off a
full-screen surface on a 424px viewport; back and a 44px close button
answer it instead.
## slskd's API, read from its source rather than a live daemon (2026-09-26)
`backend/download/provider_slskd.go` had never run against a real slskd
when #263–#272 shipped, so its assumptions were checked against slskd
0.26.0's source. One was wrong, and one design was only safe by luck.
These are properties of someone else's server; re-check on an upgrade.
`TestSlskdLive` (env-gated, see its comment) is the way to confirm them
against a running one.
- **`searchTimeout` is seconds, from the last response**, minimum 5
(`SearchRequest.cs`). We sent milliseconds (#274). The other search
options — `responseLimit`, `fileLimit`, `filterResponses`,
`minimumResponseFileCount`, `maximumPeerQueueLength` — are named as we
send them; slskd's defaults are 100 responses, 10 000 files, queue
1 000 000.
- **`GET /searches/{id}/responses` exists**, and `DELETE
/transfers/downloads/{user}/{id}?remove=true` cancels and removes.
- **A finished download is moved to `<downloads>/<Subdirectory>/`**,
where `Destination.Subdirectory` defaults to `${SOURCE_DIRECTORY}`
(the remote leaf folder) and is user-configurable. A taken name is
written as `name_<ticks>.ext` (`Destination.Exists = rename`, the
default). No transfer record says where the file went.
- **Batch enqueue (`POST /transfers/downloads/batches`) is new in 0.26.0**
and is the only way to choose where a file lands: `options.destination`
overrides the subdirectory pattern. A batch's files land flat in it,
so one batch per disc. On an older daemon that path is routed to the
per-user enqueue as username "batches" and the object body is
rejected with 400 — which is why 400 means "no batches" here.
- **`GET .../downloads/{user}/{id}/position` asks the peer** and returns
a bare integer. slskd's own comment on `PlaceInQueue` is "may be
wildly innacurate to the point of uselessness", which is why #275 acts
only on two readings in a row.
+331 -4001
View File
File diff suppressed because it is too large Load Diff
+1 -1
View File
@@ -13,7 +13,7 @@ frontend, bridged by [Wails v3](https://wails.io/).
| Tool | Version | | Tool | Version |
|------|---------| |------|---------|
| Go | 1.25+ | | Go | 1.26+ |
| Node.js | 22+ | | Node.js | 22+ |
| pnpm | 10+ | | pnpm | 10+ |
| Wails CLI | v3 — vendored, no install needed (`go tool wails3`) | | Wails CLI | v3 — vendored, no install needed (`go tool wails3`) |
+26 -6
View File
@@ -4,6 +4,7 @@ import (
"bytes" "bytes"
"context" "context"
"encoding/json" "encoding/json"
"errors"
"fmt" "fmt"
"io" "io"
"net/http" "net/http"
@@ -144,17 +145,36 @@ func (c *apiClient) checkStatus(resp *http.Response) error {
case resp.StatusCode >= 400: case resp.StatusCode >= 400:
snippet, _ := io.ReadAll(io.LimitReader(resp.Body, 512)) snippet, _ := io.ReadAll(io.LimitReader(resp.Body, 512))
return fmt.Errorf( return fmt.Errorf("%w: %w", c.errUnreachable, &httpStatusError{
"%w: HTTP %d: %s", code: resp.StatusCode,
c.errUnreachable, body: strings.TrimSpace(string(snippet)),
resp.StatusCode, })
strings.TrimSpace(string(snippet)),
)
default: default:
return nil return nil
} }
} }
// httpStatusError is a non-2xx answer, kept typed so a caller can tell
// a missing endpoint from a daemon that is down.
type httpStatusError struct {
code int
body string
}
func (e *httpStatusError) Error() string {
return fmt.Sprintf("HTTP %d: %s", e.code, e.body)
}
// statusCode returns the HTTP status an error carries, or 0.
func statusCode(err error) int {
var se *httpStatusError
if errors.As(err, &se) {
return se.code
}
return 0
}
// decodeJSON decodes a JSON string into out. Providers whose auth or // decodeJSON decodes a JSON string into out. Providers whose auth or
// response handling does not fit apiClient still parse bodies the same // response handling does not fit apiClient still parse bodies the same
// way, so the helper lives here rather than being repeated. // way, so the helper lives here rather than being repeated.
+248
View File
@@ -0,0 +1,248 @@
package download
import (
"context"
"os"
"path/filepath"
"testing"
)
// Multi-disc rips, single-track results and coverage counted in tracks
// rather than files (#270).
func TestParsePathReadsTheDiscFromItsFolder(t *testing.T) {
t.Parallel()
cases := []struct {
path string
disc int
track int
folder string
}{
{`\share\Pink Floyd - The Wall (1979)\CD2\03 Hey You.flac`, 2, 3, "The Wall"},
{`\share\The Wall\Disc 1\01 In The Flesh.flac`, 1, 1, "The Wall"},
{`\share\The Wall\[Disk-2]\01 Hey You.flac`, 2, 1, "The Wall"},
{`\share\The Wall\CD1 - Live\04 Mother.flac`, 1, 4, "The Wall"},
// The filename's own disc number is more specific than the folder.
{`\share\The Wall\CD1\2-05 Comfortably Numb.flac`, 2, 5, "The Wall"},
// Not a disc folder: a number is required.
{`\share\CDs\The Wall\01 In The Flesh.flac`, 0, 1, "The Wall"},
}
for _, tc := range cases {
t.Run(tc.path, func(t *testing.T) {
t.Parallel()
got := ParsePath(tc.path)
if got.Disc != tc.disc || got.Track != tc.track || got.Folder != tc.folder {
t.Errorf(
"ParsePath = disc %d track %d folder %q, want %d %d %q",
got.Disc, got.Track, got.Folder, tc.disc, tc.track, tc.folder,
)
}
})
}
}
func TestAlbumDir(t *testing.T) {
t.Parallel()
cases := map[string]string{
`\share\Album\CD1\01 A.flac`: "/share/Album",
`\share\Album\01 A.flac`: "/share/Album",
`CD1\01 A.flac`: "CD1",
`\share\CD Collection\01.mp3`: "/share/CD Collection",
}
for in, want := range cases {
if got := AlbumDir(in); got != want {
t.Errorf("AlbumDir(%q) = %q, want %q", in, got, want)
}
}
}
// One album shared as CD1/CD2 is one candidate, named after the album.
func TestSlskdGroupsDiscFoldersIntoOneCandidate(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
stub.responses = []slskdResponse{{
Username: "peer",
Files: []slskdFile{
{Filename: `\m\The Wall\CD1\01 In The Flesh.flac`, Size: 1},
{Filename: `\m\The Wall\CD1\02 The Thin Ice.flac`, Size: 1},
{Filename: `\m\The Wall\CD2\01 Hey You.flac`, Size: 1},
{Filename: `\m\The Wall\CD2\02 Is There Anybody Out There.flac`, Size: 1},
},
}}
s, _ := newStubSlskd(t, stub)
got, err := s.Search(context.Background(), Download{Query: "the wall"})
if err != nil {
t.Fatalf("Search: %v", err)
}
if len(got) != 1 {
t.Fatalf("got %d candidates, want the two discs as one", len(got))
}
if got[0].Title != "The Wall" || len(got[0].Files) != 4 {
t.Errorf(
"candidate = %q with %d files, want \"The Wall\" with 4",
got[0].Title, len(got[0].Files),
)
}
}
// A track search matches one file per folder, so a single-track request
// must accept a one-file folder that an album request rightly drops.
func TestSlskdKeepsASingleFileForATrackRequest(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
stub.responses = []slskdResponse{{
Username: "peer",
Files: []slskdFile{
{Filename: `\m\OK Computer\02 Paranoid Android.flac`, Size: 1},
},
}}
s, _ := newStubSlskd(t, stub)
track, err := s.Search(context.Background(), Download{
RecordingMBID: "rec-1", Artist: "Radiohead", Album: "Paranoid Android",
})
if err != nil {
t.Fatalf("Search: %v", err)
}
if len(track) != 1 {
t.Errorf("track request: got %d candidates, want 1", len(track))
}
album, err := s.Search(context.Background(), Download{
ReleaseMBID: "rel-1", Artist: "Radiohead", Album: "OK Computer",
})
if err != nil {
t.Fatalf("Search: %v", err)
}
if len(album) != 0 {
t.Errorf("album request: got %d candidates, want the one-file folder dropped", len(album))
}
}
// Two discs with a file of the same name both reach staging, each under
// its disc folder, where the importer reads the disc number from.
func TestSlskdCollectKeepsDiscFolders(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
s, downloads := newStubSlskd(t, stub)
for _, disc := range []string{"CD1", "CD2"} {
dir := filepath.Join(downloads, disc)
if err := os.MkdirAll(dir, 0o750); err != nil {
t.Fatalf("mkdir: %v", err)
}
if err := os.WriteFile(
filepath.Join(dir, "01 Intro.flac"), []byte(disc), 0o600,
); err != nil {
t.Fatalf("write: %v", err)
}
}
dst := t.TempDir()
got, err := s.collect(Candidate{Files: []CandidateFile{
{Path: `\m\Album\CD1\01 Intro.flac`, IsAudio: true},
{Path: `\m\Album\CD2\01 Intro.flac`, IsAudio: true},
}}, dst, "", nil)
if err != nil {
t.Fatalf("collect: %v", err)
}
if len(got.Files) != 2 {
t.Fatalf("collected %d files, want 2", len(got.Files))
}
for _, disc := range []string{"CD1", "CD2"} {
data, err := os.ReadFile(filepath.Join(dst, disc, "01 Intro.flac"))
if err != nil || string(data) != disc {
t.Errorf("%s's file missing or overwritten: %q, %v", disc, data, err)
}
if hint := ParsePath(filepath.Join(dst, disc, "01 Intro.flac")); hint.Disc == 0 {
t.Errorf("staged %s file lost its disc number", disc)
}
}
}
// A two-disc release whose discs both number from 01 aligns completely
// once the disc comes from the folder; before, disc 2's 01 collided with
// disc 1's.
func TestMultiDiscCandidateAlignsEveryTrack(t *testing.T) {
t.Parallel()
dl := Download{
ReleaseMBID: "the-wall",
Artist: "Pink Floyd",
Album: "The Wall",
Expected: []ExpectedTrack{
{DiscNumber: 1, Position: 1, Title: "In the Flesh?"},
{DiscNumber: 1, Position: 2, Title: "The Thin Ice"},
{DiscNumber: 2, Position: 1, Title: "Hey You"},
{DiscNumber: 2, Position: 2, Title: "Is There Anybody Out There?"},
},
}
c := Candidate{
Title: "The Wall",
Files: []CandidateFile{
{Path: `\m\Pink Floyd - The Wall\CD1\01 In the Flesh.flac`, Size: 1},
{Path: `\m\Pink Floyd - The Wall\CD1\02 The Thin Ice.flac`, Size: 1},
{Path: `\m\Pink Floyd - The Wall\CD2\01 Hey You.flac`, Size: 1},
{Path: `\m\Pink Floyd - The Wall\CD2\02 Is There Anybody Out There.flac`, Size: 1},
},
}
got := Score(dl, c, 50, AutoDownloadPrefs{})
if got.Match.Completeness != 1 {
t.Errorf("completeness = %f, want 1", got.Match.Completeness)
}
if got.Match.AlbumFit < 0.99 {
t.Errorf("album fit = %f, want the album's own name to match", got.Match.AlbumFit)
}
if got.Match.Overall < minMatch {
t.Errorf("match = %f, want it to clear the auto-pick bar %f", got.Match.Overall, minMatch)
}
}
// Ten files against a ten-track album is not a complete album when only
// three of them are its tracks.
func TestCompletenessCountsTracksNotFiles(t *testing.T) {
t.Parallel()
dl := okComputer()
c := Candidate{Title: "OK Computer", Files: []CandidateFile{
{Path: `\m\Radiohead - OK Computer\Airbag.flac`, Size: 1},
{Path: `\m\Radiohead - OK Computer\Paranoid Android.flac`, Size: 1},
{Path: `\m\Radiohead - OK Computer\Exit Music (For a Film).flac`, Size: 1},
{Path: `\m\Radiohead - OK Computer\Creep.flac`, Size: 1},
}}
got := Score(dl, c, 50, AutoDownloadPrefs{})
if got.Match.Completeness > 0.76 {
t.Errorf(
"completeness = %f with 3 of 4 tracks present, want at most 0.75",
got.Match.Completeness,
)
}
}
+13 -12
View File
@@ -47,7 +47,7 @@ func grabAll(
go func() { go func() {
defer wg.Done() defer wg.Done()
f.manager.grab(ctx, dl, candidate, nil) f.manager.grab(ctx, dl, candidate, nil, false)
}() }()
} }
@@ -79,9 +79,9 @@ func TestConcurrencyForPrefersOverrideThenKind(t *testing.T) {
want int want int
}{ }{
{ {
name: "slskd defaults to one", name: "slskd defaults to a few peers",
cfg: Config{Kind: KindSlskd}, cfg: Config{Kind: KindSlskd},
want: 1, want: 3,
}, },
{ {
name: "usenet defaults higher", name: "usenet defaults higher",
@@ -92,9 +92,9 @@ func TestConcurrencyForPrefersOverrideThenKind(t *testing.T) {
name: "explicit override wins", name: "explicit override wins",
cfg: Config{ cfg: Config{
Kind: KindSlskd, Kind: KindSlskd,
Settings: map[string]string{concurrencyKey: "3"}, Settings: map[string]string{concurrencyKey: "1"},
}, },
want: 3, want: 1,
}, },
{ {
name: "nonsense override falls back", name: "nonsense override falls back",
@@ -102,7 +102,7 @@ func TestConcurrencyForPrefersOverrideThenKind(t *testing.T) {
Kind: KindSlskd, Kind: KindSlskd,
Settings: map[string]string{concurrencyKey: "not a number"}, Settings: map[string]string{concurrencyKey: "not a number"},
}, },
want: 1, want: 3,
}, },
{ {
name: "zero override falls back", name: "zero override falls back",
@@ -110,7 +110,7 @@ func TestConcurrencyForPrefersOverrideThenKind(t *testing.T) {
Kind: KindSlskd, Kind: KindSlskd,
Settings: map[string]string{concurrencyKey: "0"}, Settings: map[string]string{concurrencyKey: "0"},
}, },
want: 1, want: 3,
}, },
{ {
name: "unknown kind falls back to the global default", name: "unknown kind falls back to the global default",
@@ -126,9 +126,9 @@ func TestConcurrencyForPrefersOverrideThenKind(t *testing.T) {
} }
} }
// The reason the per-provider cap exists: a Soulseek daemon capped at // The reason the per-provider cap exists: a daemon capped at one
// one transfer must serialize, even when the global cap would allow // transfer must serialize, even when the global cap would allow more and
// more and the user has queued several albums at once. // the user has queued several albums at once.
func TestPerProviderCapSerializesTransfers(t *testing.T) { func TestPerProviderCapSerializesTransfers(t *testing.T) {
t.Parallel() t.Parallel()
@@ -142,6 +142,7 @@ func TestPerProviderCapSerializesTransfers(t *testing.T) {
ID: 1, ID: 1,
Kind: KindSlskd, Kind: KindSlskd,
Priority: 50, Priority: 50,
Settings: map[string]string{concurrencyKey: "1"},
}, slow) }, slow)
// Three requests against the same one-at-a-time provider. // Three requests against the same one-at-a-time provider.
@@ -210,8 +211,8 @@ func TestSyncSemaphoresReplacesChangedLimits(t *testing.T) {
f.manager.installProvider(Config{ID: 1, Kind: KindSlskd}, nil) f.manager.installProvider(Config{ID: 1, Kind: KindSlskd}, nil)
first := f.manager.semaphoreFor(1) first := f.manager.semaphoreFor(1)
if cap(first) != 1 { if want := kindConcurrency[KindSlskd]; cap(first) != want {
t.Fatalf("slskd semaphore cap = %d, want 1", cap(first)) t.Fatalf("slskd semaphore cap = %d, want %d", cap(first), want)
} }
// Same limit: the semaphore is kept, so in-flight accounting is not // Same limit: the semaphore is kept, so in-flight accounting is not
+261
View File
@@ -0,0 +1,261 @@
package download
import (
"context"
"errors"
"os"
"testing"
)
// A transfer that fails on one copy of an album is not a failed
// download while another acceptable copy exists. On Soulseek the usual
// failure is one peer being offline, with several others offering the
// same folder.
var errPeerOffline = errors.New("peer went offline")
func TestManagerFallsBackToTheNextCandidate(t *testing.T) {
t.Parallel()
f := newManagerFixture(t)
// The failing source ranks first on priority, so the fallback is
// what reaches the one that works.
bad := fakeWithAlbum(1, "offline-peer", ".flac")
bad.GrabErr = errPeerOffline
good := fakeWithAlbum(2, "online-peer", ".flac")
f.manager.installProvider(Config{ID: 1, Priority: 90}, bad)
f.manager.installProvider(Config{ID: 2, Priority: 10}, good)
dl := fourTrackDownload()
if _, err := f.manager.Start(context.Background(), dl); err != nil {
t.Fatalf("Start: %v", err)
}
waitForDownloadState(t, f.store, dl.ID, StateComplete)
if bad.GrabCalls != 1 || good.GrabCalls != 1 {
t.Errorf(
"grabs: failing=%d working=%d, want 1 and 1",
bad.GrabCalls, good.GrabCalls,
)
}
// The abandoned attempt's staging goes with it; only a request that
// fails outright keeps its staging for inspection.
waitFor(t, func() bool {
entries, err := os.ReadDir(f.staging.Root())
return err == nil && len(entries) == 0
}, "the failed attempt's staging was never released")
}
// Falling back must not lower the bar. A second choice outside the
// user's guardrails is not a choice auto-pick may make, first or second.
func TestManagerFallbackRespectsTheGuardrails(t *testing.T) {
t.Parallel()
f := newManagerFixture(t)
f.manager.SetPreferences(AutoDownloadPrefs{MaxSizeMB: 50})
bad := fakeWithAlbum(1, "offline-peer", ".flac")
bad.GrabErr = errPeerOffline
bad.Candidates[0].TotalSize = 40 << 20
huge := fakeWithAlbum(2, "oversized", ".flac")
huge.Candidates[0].TotalSize = 900 << 20
f.manager.installProvider(Config{ID: 1, Priority: 90}, bad)
f.manager.installProvider(Config{ID: 2, Priority: 10}, huge)
dl := fourTrackDownload()
if _, err := f.manager.Start(context.Background(), dl); err != nil {
t.Fatalf("Start: %v", err)
}
waitForDownloadState(t, f.store, dl.ID, StateFailed)
if huge.GrabCalls != 0 {
t.Errorf("fell back to a candidate over the size ceiling")
}
}
// A copy the user picked by hand is the copy they asked for. Quietly
// substituting another is a decision they did not make.
func TestManagerPickDoesNotFallBack(t *testing.T) {
t.Parallel()
f := newManagerFixture(t)
bad := fakeWithAlbum(1, "offline-peer", ".flac")
bad.GrabErr = errPeerOffline
good := fakeWithAlbum(2, "online-peer", ".flac")
f.manager.installProvider(Config{ID: 1, Priority: 90}, bad)
f.manager.installProvider(Config{ID: 2, Priority: 10}, good)
// A ceiling below both copies parks the result set for the user.
f.manager.SetPreferences(AutoDownloadPrefs{MaxSizeMB: 1})
bad.Candidates[0].TotalSize = 30 << 20
good.Candidates[0].TotalSize = 30 << 20
dl := fourTrackDownload()
if _, err := f.manager.Start(context.Background(), dl); err != nil {
t.Fatalf("Start: %v", err)
}
if err := f.manager.Pick(
context.Background(), dl.ID, "offline-peer-cand",
); err != nil {
t.Fatalf("Pick: %v", err)
}
waitForDownloadState(t, f.store, dl.ID, StateFailed)
if good.GrabCalls != 0 {
t.Errorf("a hand-picked grab fell back to another candidate")
}
}
// Fallback is for surviving an offline peer or two, not for walking a
// forty-peer list for six hours.
func TestManagerFallbackIsBounded(t *testing.T) {
t.Parallel()
f := newManagerFixture(t)
var providers []*FakeProvider
for i := int64(1); i <= maxGrabAttempts+2; i++ {
p := fakeWithAlbum(i, "peer-"+itoa(int(i)), ".flac")
p.GrabErr = errPeerOffline
f.manager.installProvider(Config{ID: i, Priority: 50}, p)
providers = append(providers, p)
}
dl := fourTrackDownload()
if _, err := f.manager.Start(context.Background(), dl); err != nil {
t.Fatalf("Start: %v", err)
}
waitForDownloadState(t, f.store, dl.ID, StateFailed)
grabs := 0
for _, p := range providers {
grabs += p.GrabCalls
}
if grabs != maxGrabAttempts {
t.Errorf("grabs = %d, want %d", grabs, maxGrabAttempts)
}
}
// The veto judges the best candidate *inside* the guardrails, so the
// grab has to take that one — not the overall best, which may be the
// very copy the user said not to take unattended.
func TestManagerAutoPickTakesTheBestEligibleCandidate(t *testing.T) {
t.Parallel()
f := newManagerFixture(t)
f.manager.SetPreferences(AutoDownloadPrefs{MaxSizeMB: 50})
huge := fakeWithAlbum(1, "oversized", ".flac")
huge.Candidates[0].TotalSize = 900 << 20
fits := fakeWithAlbum(2, "fits", ".flac")
fits.Candidates[0].TotalSize = 40 << 20
f.manager.installProvider(Config{ID: 1, Priority: 90}, huge)
f.manager.installProvider(Config{ID: 2, Priority: 10}, fits)
dl := fourTrackDownload()
ranked, err := f.manager.Start(context.Background(), dl)
if err != nil {
t.Fatalf("Start: %v", err)
}
if ranked[0].ID != "oversized-cand" {
t.Fatalf("fixture: best overall is %s, want the oversized copy", ranked[0].ID)
}
waitForDownloadState(t, f.store, dl.ID, StateComplete)
if huge.GrabCalls != 0 || fits.GrabCalls != 1 {
t.Errorf(
"grabs: oversized=%d fits=%d, want 0 and 1",
huge.GrabCalls, fits.GrabCalls,
)
}
}
// On Soulseek a failure is the peer's, so every folder that peer offered
// goes with it. Elsewhere a failure is the release's, and one indexer's
// other releases are still worth trying.
func TestRuledOutBy(t *testing.T) {
t.Parallel()
failed := []Candidate{
{ID: "slskd:alice:Album", Kind: KindSlskd, ProviderID: 1, Origin: "alice"},
{ID: "tracker-1", Kind: KindProwlarr, ProviderID: 2, Origin: "indexer"},
}
cases := []struct {
name string
c Candidate
want bool
}{
{
name: "the same candidate",
c: Candidate{ID: "tracker-1", Kind: KindProwlarr, ProviderID: 2, Origin: "indexer"},
want: true,
},
{
name: "another folder from a failed peer",
c: Candidate{
ID: "slskd:alice:Album (2)",
Kind: KindSlskd,
ProviderID: 1,
Origin: "alice",
},
want: true,
},
{
name: "another peer",
c: Candidate{ID: "slskd:bob:Album", Kind: KindSlskd, ProviderID: 1, Origin: "bob"},
want: false,
},
{
name: "another release from the same indexer",
c: Candidate{ID: "tracker-2", Kind: KindProwlarr, ProviderID: 2, Origin: "indexer"},
want: false,
},
{
name: "a peer of the same name on a different daemon",
c: Candidate{
ID: "slskd:alice:Album",
Kind: KindSlskd,
ProviderID: 3,
Origin: "alice",
},
want: false,
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
if got := ruledOutBy(tc.c, failed); got != tc.want {
t.Errorf("ruledOutBy = %v, want %v", got, tc.want)
}
})
}
}
+190
View File
@@ -0,0 +1,190 @@
package download
import (
"cmp"
"context"
"fmt"
"strings"
"yellowjacket/backend/jobs"
)
// Filling in an almost-complete album (#276).
//
// A grab that delivers nine of twelve tracks clears the completeness
// floor and is imported, and before this the other three were never
// looked for. On Soulseek that is the commonest way an album ends up
// almost right: one peer's folder is missing a track, or one file
// failed. So after an import, each missing track is searched for on
// its own and fetched from somewhere else, into the same album.
// maxFillInTracks bounds how many tracks are fetched one by one. An
// album missing more than a few is a different candidate's job, not a
// dozen single-track grabs.
const maxFillInTracks = 3
// fillIn fetches the tracks a successful import did not deliver. It
// never fails the download: the album is already imported, and a track
// it cannot find is logged and left.
func (m *Manager) fillIn(
ctx context.Context,
dl Download,
main Candidate,
imported ImportResult,
job *jobs.Handle,
) {
missing := missingTracks(dl, imported.Matched)
if len(missing) == 0 {
return
}
for _, t := range missing {
if ctx.Err() != nil {
return
}
paths, err := m.fillInTrack(ctx, dl, main, t, job)
if err != nil {
m.logger.Info(
"could not fill in a missing track",
"download", dl.ID,
"track", t.Title,
"error", err,
)
if job != nil {
job.Logf(jobs.LevelWarn, fmt.Sprintf(
"Could not find %q elsewhere: %v", t.Title, err,
))
}
continue
}
if job != nil {
job.Logf(jobs.LevelInfo, fmt.Sprintf(
"Filled in %q from another source (%d file)", t.Title, len(paths),
))
}
}
}
// missingTracks is what an import left out, when filling it in is
// worth trying: an album with a tracklist, a few tracks short.
func missingTracks(dl Download, matched []ExpectedTrack) []ExpectedTrack {
// A recording request is one track; there is no album to complete.
if dl.RecordingMBID != "" || len(dl.Expected) < 2 {
return nil
}
have := make(map[trackKey]bool, len(matched))
for _, t := range matched {
have[keyOf(t)] = true
}
var missing []ExpectedTrack
for _, t := range dl.Expected {
if !have[keyOf(t)] {
missing = append(missing, t)
}
}
// A half-empty album was a poor copy, not a nearly complete one.
if len(missing) > maxFillInTracks || 2*len(missing) >= len(dl.Expected) {
return nil
}
return missing
}
// fillInTrack searches for one track and grabs the first acceptable
// copy that is not from the source that already failed to supply it.
// One attempt: a fill-in that walks a candidate list per track would
// multiply a download's grabs by the number of gaps.
func (m *Manager) fillInTrack(
ctx context.Context,
dl Download,
main Candidate,
t ExpectedTrack,
job *jobs.Handle,
) ([]string, error) {
want := trackRequest(dl, t)
ranked, err := m.Search(ctx, want)
if err != nil {
return nil, err
}
prefs := m.preferences()
for _, c := range ranked {
if ruledOutBy(c, []Candidate{main}) || !autoAcceptable(want, c, prefs) {
continue
}
// The track is being put into an album this app placed; a
// delegate would put it in its own library instead.
if plan, err := m.planTransfer(want, c); err != nil || plan.delegated() {
continue
}
narrowed, ok := narrowTo(c, t)
if !ok {
continue
}
out := m.attemptGrab(ctx, dl, narrowed, job, []ExpectedTrack{t})
if out.item.StagingDir != "" {
if err := m.staging.Release(out.item.StagingDir); err != nil {
m.logger.Warn("could not release staging dir", "error", err)
}
}
if out.err != nil {
return nil, out.err
}
if err := m.store.SetItemImported(
ctx, out.item.ID, out.imported.Paths,
); err != nil {
m.logger.Warn("could not record imported paths", "error", err)
}
return out.imported.Paths, nil
}
return nil, ErrNoCandidates
}
// trackRequest is the search for one track of an album: the track's
// artist and title as the query, the album kept so a copy from that
// album outranks the same song off a compilation, and one expected
// track so a single file is a complete answer.
func trackRequest(dl Download, t ExpectedTrack) Download {
artist := cmp.Or(t.Artist, dl.Artist)
want := dl
want.Query = strings.TrimSpace(artist + " " + t.Title)
want.Expected = []ExpectedTrack{t}
return want
}
// narrowTo trims a candidate to the one file that aligns to t, so the
// grab fetches a track rather than whatever else the folder offered.
func narrowTo(c Candidate, t ExpectedTrack) (Candidate, bool) {
aligned, _ := matchFiles(c.Files, []ExpectedTrack{t})
for _, f := range aligned {
if f.IsAudio && f.MatchedTo == t.Position {
c.Files = []CandidateFile{f}
c.TotalSize = f.Size
return c, true
}
}
return Candidate{}, false
}
+168
View File
@@ -0,0 +1,168 @@
package download
import (
"context"
"errors"
"os"
"path/filepath"
"testing"
)
// Filling in the tracks an almost-complete album is missing (#276).
func fiveTrackTitles() []string {
return append(allTitles(), "Let Down")
}
func fiveTrackDownload() Download {
dl := fourTrackDownload()
dl.Expected = append(dl.Expected, ExpectedTrack{Position: 5, Title: "Let Down"})
return dl
}
func TestMissingTracks(t *testing.T) {
t.Parallel()
dl := fiveTrackDownload()
got := func(positions ...int) []ExpectedTrack {
out := make([]ExpectedTrack, 0, len(positions))
for _, p := range positions {
out = append(out, dl.Expected[p-1])
}
return out
}
cases := []struct {
name string
dl Download
matched []ExpectedTrack
want int
}{
{"complete", dl, got(1, 2, 3, 4, 5), 0},
{"one short", dl, got(1, 2, 3, 4), 1},
{"two short", dl, got(1, 2, 3), 2},
{"half gone is a poor copy", dl, got(1, 2), 0},
{"a recording is not an album", func() Download {
d := dl
d.RecordingMBID = "rec"
return d
}(), got(1, 2, 3, 4), 0},
}
for _, tc := range cases {
if n := len(missingTracks(tc.dl, tc.matched)); n != tc.want {
t.Errorf("%s: %d missing, want %d", tc.name, n, tc.want)
}
}
big := fiveTrackDownload()
for i := 6; i <= 20; i++ {
big.Expected = append(big.Expected, ExpectedTrack{Position: i, Title: "T" + itoa(i)})
}
if n := len(missingTracks(big, big.Expected[:16])); n != 0 {
t.Errorf("four of twenty missing: %d filled in, want none past %d", n, maxFillInTracks)
}
}
// A fill-in import takes only the file that is the missing track, and
// places it in the album with the album's tags; anything else the grab
// brought is left out.
func TestImportOnlyTakesTheMissingTrack(t *testing.T) {
t.Parallel()
f := newImportFixture(t,
"05 - Let Down.flac",
"02 - Paranoid Android.flac",
)
dl := fiveTrackDownload()
got, err := f.importer.Import(
context.Background(), dl,
Result{Dir: f.dir, Files: f.files},
ImportOptions{LibraryRoot: f.root, WriteTags: true, Only: dl.Expected[4:]},
)
if err != nil {
t.Fatalf("Import: %v", err)
}
want := filepath.Join(f.root, "Radiohead", "OK Computer", "05 Let Down.flac")
if len(got.Paths) != 1 || got.Paths[0] != want {
t.Errorf("imported %q, want only %s", got.Paths, want)
}
// A file that is not the missing track is not imported at all.
g := newImportFixture(t, "02 - Paranoid Android.flac")
if _, err := g.importer.Import(
context.Background(), dl,
Result{Dir: g.dir, Files: g.files},
ImportOptions{LibraryRoot: g.root, WriteTags: true, Only: dl.Expected[4:]},
); !errors.Is(err, ErrTooIncomplete) {
t.Errorf("Import = %v, want nothing matched", err)
}
}
// The album comes from one source missing its fifth track; the fifth is
// then found on its own at another and lands in the same album.
func TestManagerFillsInAMissingTrack(t *testing.T) {
t.Parallel()
f := newManagerFixture(t)
titles := fiveTrackTitles()
album := NewFakeProvider(1, "album", Caps{CanSearch: true, CanTransport: true})
ac := candidateFor("album-cand", titles, ".flac", 30_000_000)
ac.ProviderID = 1
album.Candidates = []Candidate{ac}
for i, tt := range titles[:4] {
album.Written[trackToken(i+1)+" - "+tt+".flac"] = []byte("audio-data")
}
single := NewFakeProvider(2, "single", Caps{CanSearch: true, CanTransport: true})
sc := Candidate{
ID: "single-cand",
Protocol: ProtocolDirect,
Title: "Radiohead - OK Computer",
Artist: "Radiohead",
Files: []CandidateFile{{
Path: "Radiohead - OK Computer/05 - Let Down.flac",
Size: 30_000_000,
}},
Health: 0.5,
ProviderID: 2,
}
single.Candidates = []Candidate{sc}
single.Written["05 - Let Down.flac"] = []byte("audio-data")
f.manager.installProvider(Config{ID: 1, Priority: 90}, album)
f.manager.installProvider(Config{ID: 2, Priority: 10}, single)
dl := fiveTrackDownload()
if _, err := f.manager.Start(context.Background(), dl); err != nil {
t.Fatalf("Start: %v", err)
}
waitForDownloadState(t, f.store, dl.ID, StateComplete)
if album.GrabCallCount() != 1 || single.GrabCallCount() != 1 {
t.Errorf(
"grabs: album=%d single=%d, want 1 and 1",
album.GrabCallCount(), single.GrabCallCount(),
)
}
for i, tt := range titles {
p := filepath.Join(f.root, "Radiohead", "OK Computer", trackToken(i+1)+" "+tt+".flac")
if _, err := os.Stat(p); err != nil {
t.Errorf("track %d not in the library: %v", i+1, err)
}
}
}
+61 -2
View File
@@ -79,6 +79,12 @@ type ImportOptions struct {
// them. Off for delegate providers, which have already imported // them. Off for delegate providers, which have already imported
// and tagged the files themselves. // and tagged the files themselves.
WriteTags bool WriteTags bool
// Only, when set, imports just the files that align to these
// tracks of the download and skips the completeness check: it is a
// fill-in for tracks an earlier grab of the same album did not
// deliver (#276), tagged and placed as part of that album.
Only []ExpectedTrack
} }
// DefaultPathTemplate is the layout used when none is configured. // DefaultPathTemplate is the layout used when none is configured.
@@ -118,6 +124,10 @@ type ImportResult struct {
// Skipped counts non-audio files left in staging (logs, cue sheets, // Skipped counts non-audio files left in staging (logs, cue sheets,
// scene .nfo files) — deliberately not imported. // scene .nfo files) — deliberately not imported.
Skipped int Skipped int
// Matched are the expected tracks an imported file was aligned to,
// which is how a caller learns what the grab did not deliver.
Matched []ExpectedTrack
} }
// Import verifies, tags and moves a completed grab into the library. // Import verifies, tags and moves a completed grab into the library.
@@ -140,14 +150,25 @@ func (i *Importer) Import(
return ImportResult{}, ErrNoAudio return ImportResult{}, ErrNoAudio
} }
if err := checkCompleteness(len(audio), dl); err != nil { if len(opts.Only) == 0 {
return ImportResult{}, err if err := checkCompleteness(len(audio), dl); err != nil {
return ImportResult{}, err
}
} }
// Align staged files to the expected tracklist so tags and // Align staged files to the expected tracklist so tags and
// filenames reflect the release, not the uploader's naming. // filenames reflect the release, not the uploader's naming.
plan := i.planFiles(audio, dl) plan := i.planFiles(audio, dl)
if len(opts.Only) > 0 {
plan = onlyTracks(plan, opts.Only)
if len(plan) == 0 {
return ImportResult{}, fmt.Errorf(
"%w: no file matched the missing track", ErrTooIncomplete,
)
}
}
out := ImportResult{ out := ImportResult{
Paths: make([]string, 0, len(plan)), Paths: make([]string, 0, len(plan)),
Skipped: skipped, Skipped: skipped,
@@ -184,11 +205,49 @@ func (i *Importer) Import(
} }
out.Paths = append(out.Paths, dest) out.Paths = append(out.Paths, dest)
if p.Matched {
out.Matched = append(out.Matched, p.Track)
}
} }
return out, nil return out, nil
} }
// trackKey identifies an expected track within a release.
type trackKey struct{ disc, position int }
func keyOf(t ExpectedTrack) trackKey {
return trackKey{disc: t.DiscNumber, position: t.Position}
}
// onlyTracks keeps the planned files aligned to one of want. A fill-in
// grab can bring more than the one file it was after — a folder where
// the title also matched a live take — and anything else would land in
// the album as a duplicate or a stranger.
func onlyTracks(plan []plannedFile, want []ExpectedTrack) []plannedFile {
keys := make(map[trackKey]bool, len(want))
for _, t := range want {
keys[keyOf(t)] = true
}
out := make([]plannedFile, 0, len(want))
seen := map[trackKey]bool{}
for _, p := range plan {
k := keyOf(p.Track)
if !p.Matched || !keys[k] || seen[k] {
continue
}
seen[k] = true
out = append(out, p)
}
return out
}
// plannedFile pairs a staged file with the expected track it matched. // plannedFile pairs a staged file with the expected track it matched.
type plannedFile struct { type plannedFile struct {
Source string Source string
+77
View File
@@ -0,0 +1,77 @@
package download
import (
"context"
"sync"
)
// keyedLock is a set of mutexes created on demand, one per key, that
// honour a context while waiting. An entry lives only while someone
// holds or waits on it, so a key per Soulseek peer or per folder name
// does not accumulate for the life of the process.
type keyedLock[K comparable] struct {
mu sync.Mutex
held map[K]*keyedEntry
}
type keyedEntry struct {
ch chan struct{}
// refs counts holders and waiters; the entry is dropped at zero.
refs int
}
// acquire blocks until k is free or ctx ends, and returns the function
// that frees it.
func (l *keyedLock[K]) acquire(ctx context.Context, k K) (func(), error) {
l.mu.Lock()
if l.held == nil {
l.held = map[K]*keyedEntry{}
}
e, ok := l.held[k]
if !ok {
e = &keyedEntry{ch: make(chan struct{}, 1)}
l.held[k] = e
}
e.refs++
l.mu.Unlock()
select {
case e.ch <- struct{}{}:
case <-ctx.Done():
l.drop(k, e)
return nil, ctx.Err()
}
var once sync.Once
return func() {
once.Do(func() {
<-e.ch
l.drop(k, e)
})
}, nil
}
func (l *keyedLock[K]) drop(k K, e *keyedEntry) {
l.mu.Lock()
defer l.mu.Unlock()
e.refs--
if e.refs == 0 {
delete(l.held, k)
}
}
// size reports how many keys are held or awaited, for tests.
func (l *keyedLock[K]) size() int {
l.mu.Lock()
defer l.mu.Unlock()
return len(l.held)
}
+248 -65
View File
@@ -59,13 +59,15 @@ const concurrencyKey = "maxConcurrent"
// A single global cap is the wrong shape here: usenet and torrent // A single global cap is the wrong shape here: usenet and torrent
// clients are built to run many transfers at once and are throttled by // clients are built to run many transfers at once and are throttled by
// bandwidth, while Soulseek transfers come from one person's home // bandwidth, while Soulseek transfers come from one person's home
// upload slot. Hitting the same peer with parallel requests gets you // upload slot. Politeness there is per *peer* — asking one user for two
// queued behind everyone else at best and banned at worst, so slskd is // folders at once gets you queued behind everyone else at best and
// capped at one — the polite number, and the one that actually // banned at worst — and the manager holds that line separately, one
// completes fastest, because a Soulseek peer serves one file at a time // grab per peer (peerLocks). Two different users do not compete for
// regardless of how many you ask for. // anyone's slot, so the daemon-wide number only bounds how many peers
// are asked at once, and one slow peer no longer serialises every other
// Soulseek download behind it.
var kindConcurrency = map[Kind]int{ var kindConcurrency = map[Kind]int{
KindSlskd: 1, KindSlskd: 3,
KindYtDlp: 2, KindYtDlp: 2,
KindQBittorrent: 4, KindQBittorrent: 4,
KindSABnzbd: 4, KindSABnzbd: 4,
@@ -155,6 +157,11 @@ type Manager struct {
semMu sync.Mutex semMu sync.Mutex
provSem map[int64]chan struct{} provSem map[int64]chan struct{}
// peerLocks holds one grab per Soulseek peer, taken before any
// slot: a grab waiting for a busy peer must not sit on a provider
// slot another peer could be using.
peerLocks keyedLock[peerKey]
// delegatePoll is how often delegating managers are asked for // delegatePoll is how often delegating managers are asked for
// status. A field rather than the constant so tests can drive the // status. A field rather than the constant so tests can drive the
// full delegate flow without sleeping through it. // full delegate flow without sleeping through it.
@@ -577,12 +584,12 @@ func (m *Manager) Start(
)) ))
} }
if m.AutoPickable(dl, ranked) { if pick, ok := autoPick(dl, ranked, m.preferences()); ok {
if job != nil { if job != nil {
job.Logf(jobs.LevelInfo, "Auto-selected best candidate") job.Logf(jobs.LevelInfo, "Auto-selected best candidate")
} }
go m.grab(context.WithoutCancel(ctx), dl, ranked[0], job) go m.grab(context.WithoutCancel(ctx), dl, pick, job, true)
return ranked, nil return ranked, nil
} }
@@ -622,6 +629,13 @@ func (m *Manager) Attempt(
return false, veto, nil return false, veto, nil
} }
pick, ok := autoPick(dl, ranked, m.preferences())
if !ok {
// Unreachable while autoPick and AutoPickVeto agree; kept so a
// future divergence refuses rather than grabbing blind.
return false, "no candidate clears the auto-download bar", nil
}
if err := m.store.CreateDownload(ctx, dl); err != nil { if err := m.store.CreateDownload(ctx, dl); err != nil {
return false, "", err return false, "", err
} }
@@ -643,7 +657,7 @@ func (m *Manager) Attempt(
)) ))
} }
go m.grab(context.WithoutCancel(ctx), dl, ranked[0], job) go m.grab(context.WithoutCancel(ctx), dl, pick, job, true)
return true, "", nil return true, "", nil
} }
@@ -678,7 +692,7 @@ func (m *Manager) Pick(
job := m.startJob(dl) job := m.startJob(dl)
go m.grab(context.WithoutCancel(ctx), dl, *chosen, job) go m.grab(context.WithoutCancel(ctx), dl, *chosen, job, false)
return nil return nil
} }
@@ -702,13 +716,20 @@ func (m *Manager) Cancel(ctx context.Context, downloadID string) error {
return nil return nil
} }
// grab drives one candidate all the way to the library. It runs on its // grab drives one request all the way to the library. It runs on its
// own goroutine and owns the job from here on. // own goroutine and owns the job from here on.
//
// When fallback is set and a candidate's transfer fails, the next
// candidate that auto-pick would itself have accepted is tried in its
// place (see nextCandidate). It is set for the two unattended routes
// and not for a candidate the user picked by hand: they chose that copy,
// and quietly substituting another is a decision they did not make.
func (m *Manager) grab( func (m *Manager) grab(
ctx context.Context, ctx context.Context,
dl Download, dl Download,
c Candidate, c Candidate,
job *jobs.Handle, job *jobs.Handle,
fallback bool,
) { ) {
ctx, cancel := context.WithTimeout(ctx, grabTimeout) ctx, cancel := context.WithTimeout(ctx, grabTimeout)
defer cancel() defer cancel()
@@ -723,6 +744,101 @@ func (m *Manager) grab(
m.actMu.Unlock() m.actMu.Unlock()
}() }()
var failed []Candidate
for {
out := m.attemptGrab(ctx, dl, c, job, nil)
if out.err == nil {
m.fillIn(ctx, dl, c, out.imported, job)
m.finishGrab(ctx, dl, out.item, out.imported, job)
return
}
failed = append(failed, c)
next, ok := m.nextCandidate(ctx, dl, failed, out, fallback)
if !ok {
m.failDownload(ctx, job, dl.ID, out.err)
return
}
m.logger.Info(
"download candidate failed; trying the next",
"download", dl.ID,
"failed", c.ID,
"next", next.ID,
"error", out.err,
)
if job != nil {
job.Logf(jobs.LevelWarn, fmt.Sprintf(
"%s failed (%v); trying %s instead",
describeCandidate(c), out.err, describeCandidate(next),
))
}
// The failed attempt's staging holds at most a partial folder
// nobody is going to import, and the next attempt reserves its
// own. Only the final failure keeps its staging for inspection.
if out.item.StagingDir != "" {
if err := m.staging.Release(out.item.StagingDir); err != nil {
m.logger.Warn("could not release staging dir", "error", err)
}
}
c = next
}
}
// maxGrabAttempts bounds how many candidates one request will try. A
// popular album can have dozens of peers; the point of falling back is
// to survive the ordinary one or two that are offline, not to walk the
// whole list for six hours.
const maxGrabAttempts = 3
// peerKey names one Soulseek user on one daemon. The same username on
// two daemons is two logins and two queues.
type peerKey struct {
provider int64
peer string
}
// peerKeyFor returns the peer a candidate is fetched from, when the
// source is one where asking a peer for two things at once is rude.
func peerKeyFor(c Candidate) (peerKey, bool) {
if c.Kind != KindSlskd || c.Origin == "" {
return peerKey{}, false
}
return peerKey{provider: c.ProviderID, peer: c.Origin}, true
}
// grabOutcome is how one candidate's attempt ended.
type grabOutcome struct {
item DownloadItem
imported ImportResult
err error
// retryable reports whether another candidate might succeed where
// this one failed: the transfer failed, or delivered too little of
// the album. Anything else — no staging space, no library root, a
// tag write failing — would fail the next candidate identically.
retryable bool
}
// attemptGrab takes one candidate through transfer and import. It
// records the item's own failure, but not the download's: whether the
// download has failed is the caller's decision, since another candidate
// may yet succeed.
func (m *Manager) attemptGrab(
ctx context.Context,
dl Download,
c Candidate,
job *jobs.Handle,
only []ExpectedTrack,
) grabOutcome {
// Who will move the bytes is decided before any slot is taken, so // Who will move the bytes is decided before any slot is taken, so
// the transfer waits in its own provider's queue rather than in a // the transfer waits in its own provider's queue rather than in a
// global one. A delegate takes no slot at all: the transfer is // global one. A delegate takes no slot at all: the transfer is
@@ -731,21 +847,26 @@ func (m *Manager) grab(
// work against our budget. // work against our budget.
plan, err := m.planTransfer(dl, c) plan, err := m.planTransfer(dl, c)
if err != nil { if err != nil {
m.failDownload(ctx, job, dl.ID, err) return grabOutcome{err: err}
return
} }
if !plan.delegated() { if !plan.delegated() {
if key, ok := peerKeyFor(c); ok {
release, err := m.peerLocks.acquire(ctx, key)
if err != nil {
return grabOutcome{err: err}
}
defer release()
}
provSem := m.semaphoreFor(plan.transportID) provSem := m.semaphoreFor(plan.transportID)
select { select {
case provSem <- struct{}{}: case provSem <- struct{}{}:
defer func() { <-provSem }() defer func() { <-provSem }()
case <-ctx.Done(): case <-ctx.Done():
m.failDownload(ctx, job, dl.ID, ctx.Err()) return grabOutcome{err: ctx.Err()}
return
} }
globalSem := m.globalSem() globalSem := m.globalSem()
@@ -754,9 +875,7 @@ func (m *Manager) grab(
case globalSem <- struct{}{}: case globalSem <- struct{}{}:
defer func() { <-globalSem }() defer func() { <-globalSem }()
case <-ctx.Done(): case <-ctx.Done():
m.failDownload(ctx, job, dl.ID, ctx.Err()) return grabOutcome{err: ctx.Err()}
return
} }
} }
@@ -771,24 +890,30 @@ func (m *Manager) grab(
dir, err := m.staging.Reserve(item.ID) dir, err := m.staging.Reserve(item.ID)
if err != nil { if err != nil {
m.failDownload(ctx, job, dl.ID, err) return grabOutcome{err: err}
return
} }
item.StagingDir = dir item.StagingDir = dir
if err := m.store.CreateItem(ctx, item); err != nil { if err := m.store.CreateItem(ctx, item); err != nil {
m.failDownload(ctx, job, dl.ID, err) return grabOutcome{item: item, err: err}
}
return fail := func(err error, retryable bool) grabOutcome {
if serr := m.store.SetItemState(
ctx, item.ID, StateFailed, err.Error(),
); serr != nil {
m.logger.Warn("could not record item failure", "error", serr)
}
return grabOutcome{item: item, err: err, retryable: retryable}
} }
result, err := m.transfer(ctx, dl, item, plan, job) result, err := m.transfer(ctx, dl, item, plan, job)
if err != nil { if err != nil {
m.failItem(ctx, job, item, dl.ID, err) // A delegate's failure is the external manager's verdict on the
// whole request, not on one copy of it.
return return fail(err, !plan.delegated())
} }
m.setStates(ctx, dl.ID, item.ID, StateImporting) m.setStates(ctx, dl.ID, item.ID, StateImporting)
@@ -798,42 +923,117 @@ func (m *Manager) grab(
job.SetStages(importStages(2)) job.SetStages(importStages(2))
} }
var imported ImportResult
if result.Delegated { if result.Delegated {
// The external manager already placed and tagged these files in // The external manager already placed and tagged these files in
// its own library. Moving them out from under a system that is // its own library. Moving them out from under a system that is
// still managing them would be worse than useless, so the files // still managing them would be worse than useless, so the files
// are recorded where they are and the library scan picks them // are recorded where they are and the library scan picks them
// up in place. // up in place.
imported = ImportResult{Paths: result.Files}
if job != nil { if job != nil {
job.Logf(jobs.LevelInfo, fmt.Sprintf( job.Logf(jobs.LevelInfo, fmt.Sprintf(
"External manager imported %d files; recording them in place", "External manager imported %d files; recording them in place",
len(result.Files), len(result.Files),
)) ))
} }
} else {
opts := m.importOptions()
opts.WriteTags = true
opts.LibraryRoot, err = m.library.LibraryPath(dl.LibraryID) return grabOutcome{
if err != nil { item: item,
m.failItem(ctx, job, item, dl.ID, imported: ImportResult{Paths: result.Files},
fmt.Errorf("resolve library root: %w", err))
return
}
imported, err = m.importer.Import(ctx, dl, result, opts)
if err != nil {
m.failItem(ctx, job, item, dl.ID, err)
return
} }
} }
opts := m.importOptions()
opts.WriteTags = true
opts.Only = only
opts.LibraryRoot, err = m.library.LibraryPath(dl.LibraryID)
if err != nil {
return fail(fmt.Errorf("resolve library root: %w", err), false)
}
imported, err := m.importer.Import(ctx, dl, result, opts)
if err != nil {
return fail(err, errors.Is(err, ErrTooIncomplete))
}
return grabOutcome{item: item, imported: imported}
}
// nextCandidate picks the candidate to try after the ones in failed.
//
// It only ever offers a candidate auto-pick would have taken on its own
// (autoAcceptable), so falling back cannot lower the bar an unattended
// download is held to: the second choice has to clear the same gates
// the first did.
//
// On Soulseek a failure belongs to the *peer* — offline, refusing, or
// holding us in a queue — so every folder that peer offered is skipped
// with it. Elsewhere a failure belongs to the release, and only that
// candidate is.
func (m *Manager) nextCandidate(
ctx context.Context,
dl Download,
failed []Candidate,
out grabOutcome,
fallback bool,
) (Candidate, bool) {
if !fallback || !out.retryable || ctx.Err() != nil ||
len(failed) >= maxGrabAttempts {
return Candidate{}, false
}
m.resMu.RLock()
ranked := m.results[dl.ID]
m.resMu.RUnlock()
prefs := m.preferences()
for _, c := range ranked {
if ruledOutBy(c, failed) || !autoAcceptable(dl, c, prefs) {
continue
}
return c, true
}
return Candidate{}, false
}
// ruledOutBy reports whether a failure among failed also rules out c.
func ruledOutBy(c Candidate, failed []Candidate) bool {
for _, f := range failed {
if c.ID == f.ID && c.ProviderID == f.ProviderID {
return true
}
if c.Kind == KindSlskd && f.Kind == KindSlskd &&
c.ProviderID == f.ProviderID && c.Origin != "" &&
c.Origin == f.Origin {
return true
}
}
return false
}
// describeCandidate names a candidate for the job log.
func describeCandidate(c Candidate) string {
if c.Origin != "" {
return fmt.Sprintf("%q from %s", c.Title, c.Origin)
}
return fmt.Sprintf("%q", c.Title)
}
// finishGrab records a successful import and retires what the request
// was holding.
func (m *Manager) finishGrab(
ctx context.Context,
dl Download,
item DownloadItem,
imported ImportResult,
job *jobs.Handle,
) {
if err := m.store.SetItemImported( if err := m.store.SetItemImported(
ctx, item.ID, imported.Paths, ctx, item.ID, imported.Paths,
); err != nil { ); err != nil {
@@ -1177,23 +1377,6 @@ func (m *Manager) failDownload(
} }
} }
// failItem records an item-level failure and fails its download.
func (m *Manager) failItem(
ctx context.Context,
job *jobs.Handle,
item DownloadItem,
downloadID string,
err error,
) {
if serr := m.store.SetItemState(
ctx, item.ID, StateFailed, err.Error(),
); serr != nil {
m.logger.Warn("could not record item failure", "error", serr)
}
m.failDownload(ctx, job, downloadID, err)
}
// startJob registers the request in the background jobs panel. // startJob registers the request in the background jobs panel.
func (m *Manager) startJob(dl Download) *jobs.Handle { func (m *Manager) startJob(dl Download) *jobs.Handle {
if m.jobsReg == nil { if m.jobsReg == nil {
+140 -8
View File
@@ -66,6 +66,14 @@ var (
// separatorPattern splits "Artist - Album" style folder names. // separatorPattern splits "Artist - Album" style folder names.
separatorPattern = regexp.MustCompile(`\s+[-–—]\s+`) separatorPattern = regexp.MustCompile(`\s+[-–—]\s+`)
// discFolderPattern matches a directory that holds one disc of an
// album rather than the album: "CD1", "CD 2", "Disc 3", "Disk-1",
// "[Disc 2]", "CD1 - The Early Years". A number is required, so a
// folder merely called "CDs" is not one.
discFolderPattern = regexp.MustCompile(
`(?i)^\s*[\[(]?\s*(?:cd|disc|disk)\s*[-_.#]?\s*(\d{1,2})\b`,
)
) )
// FormatForPath returns the audio format implied by a path's extension, // FormatForPath returns the audio format implied by a path's extension,
@@ -94,16 +102,63 @@ type TrackHint struct {
Folder string Folder string
} }
// discFolder reports whether a directory name is one disc of an album,
// and which.
func discFolder(name string) (int, bool) {
m := discFolderPattern.FindStringSubmatch(name)
if m == nil {
return 0, false
}
n, err := strconv.Atoi(m[1])
if err != nil || n == 0 {
return 0, false
}
return n, true
}
// AlbumDir is the directory that holds a file's *album*: its parent,
// or its grandparent when the parent is a disc folder.
//
// Multi-disc rips are shared as `Album/CD1/…` and `Album/CD2/…`, and
// grouping candidates by the immediate parent split one album into two
// half-albums, each titled "CD1". Neither could clear the completeness
// or album-title bars, so a multi-disc release could not be auto-picked
// at all. A disc folder at the root has no album above it and is
// returned as it is.
func AlbumDir(p string) string {
dir := path.Dir(strings.ReplaceAll(p, `\`, "/"))
if _, ok := discFolder(path.Base(dir)); !ok {
return dir
}
parent := path.Dir(dir)
if parent == "." || parent == "/" || parent == "" {
return dir
}
return parent
}
// ParsePath extracts what it can from one candidate file path. // ParsePath extracts what it can from one candidate file path.
func ParsePath(p string) TrackHint { func ParsePath(p string) TrackHint {
// Soulseek paths are Windows-style; normalize before splitting. // Soulseek paths are Windows-style; normalize before splitting.
norm := strings.ReplaceAll(p, `\`, "/") norm := strings.ReplaceAll(p, `\`, "/")
base := path.Base(norm) base := path.Base(norm)
folder := path.Base(path.Dir(norm))
name := strings.TrimSuffix(base, path.Ext(base)) name := strings.TrimSuffix(base, path.Ext(base))
hint := TrackHint{Folder: cleanAlbumName(folder)} // The album's name is the album directory's, not a disc folder's,
// and the disc folder is where a multi-disc rip says which disc a
// file is on. A disc number in the filename ("2-01 …") is more
// specific and overrides it below.
hint := TrackHint{Folder: cleanAlbumName(path.Base(AlbumDir(norm)))}
if disc, ok := discFolder(path.Base(path.Dir(norm))); ok {
hint.Disc = disc
}
if m := trackNumPattern.FindStringSubmatch(name); m != nil { if m := trackNumPattern.FindStringSubmatch(name); m != nil {
if m[1] != "" { if m[1] != "" {
@@ -203,21 +258,72 @@ func AnnotateFiles(files []CandidateFile) []CandidateFile {
// matchFiles aligns a candidate's audio files to the expected tracklist // matchFiles aligns a candidate's audio files to the expected tracklist
// and returns the per-file assignment plus the mean title similarity of // and returns the per-file assignment plus the mean title similarity of
// the aligned pairs. // the aligned pairs. alignFiles is the same alignment with the
// duration evidence as well.
func matchFiles(
files []CandidateFile,
expected []ExpectedTrack,
) ([]CandidateFile, float64) {
a := alignFiles(files, expected)
return a.files, a.titleFit
}
// alignment is what aligning a candidate to a tracklist found.
type alignment struct {
files []CandidateFile
// titleFit is the mean title similarity over aligned pairs.
titleFit float64
// durationFit is the mean duration agreement over aligned pairs
// where both sides state a length, and timedPairs is how many such
// pairs there were.
durationFit float64
timedPairs int
aligned int
}
// durationAgreement scores how well a file's length matches the
// expected track's, in 0..1. Rips of the same master differ by a
// second or two of silence; a different edit, a live take or a
// truncated file differs by tens of seconds.
func durationAgreement(got, want int64) float64 {
const (
exactMillis = 3_000
wrongMillis = 30_000
)
d := got - want
if d < 0 {
d = -d
}
switch {
case d <= exactMillis:
return 1
case d >= wrongMillis:
return 0
default:
return 1 - float64(d-exactMillis)/float64(wrongMillis-exactMillis)
}
}
// alignFiles aligns a candidate's audio files to the expected tracklist.
// //
// Alignment is greedy by score rather than optimal: candidate folders // Alignment is greedy by score rather than optimal: candidate folders
// are small (a few dozen files at most) and the common cases — correct // are small (a few dozen files at most) and the common cases — correct
// track numbers, or clean "NN Title" names — are unambiguous, so the // track numbers, or clean "NN Title" names — are unambiguous, so the
// extra machinery of Hungarian assignment buys nothing here. // extra machinery of Hungarian assignment buys nothing here.
func matchFiles( func alignFiles(
files []CandidateFile, files []CandidateFile,
expected []ExpectedTrack, expected []ExpectedTrack,
) ([]CandidateFile, float64) { ) alignment {
annotated := make([]CandidateFile, len(files)) annotated := make([]CandidateFile, len(files))
copy(annotated, files) copy(annotated, files)
if len(expected) == 0 { if len(expected) == 0 {
return annotated, 0 return alignment{files: annotated}
} }
hints := make([]TrackHint, len(annotated)) hints := make([]TrackHint, len(annotated))
@@ -230,8 +336,19 @@ func matchFiles(
var ( var (
total float64 total float64
matched int matched int
durTotal float64
timed int
) )
// timing adds a pair's duration evidence when both sides state one.
timing := func(f CandidateFile, e ExpectedTrack) {
if f.LengthMillis > 0 && e.LengthMillis > 0 {
durTotal += durationAgreement(f.LengthMillis, e.LengthMillis)
timed++
}
}
// Pass 1: trust explicit track numbers when they are unique and in // Pass 1: trust explicit track numbers when they are unique and in
// range. A folder that numbers its files correctly is the strong // range. A folder that numbers its files correctly is the strong
// case, and title comparison only adds noise there. // case, and title comparison only adds noise there.
@@ -250,6 +367,8 @@ func matchFiles(
total += autotag.TitleSimilarity(hints[i].Title, expected[idx].Title) total += autotag.TitleSimilarity(hints[i].Title, expected[idx].Title)
matched++ matched++
timing(annotated[i], expected[idx])
} }
// Pass 2: title similarity for whatever is left. // Pass 2: title similarity for whatever is left.
@@ -284,13 +403,26 @@ func matchFiles(
total += bestSim total += bestSim
matched++ matched++
timing(annotated[i], expected[bestIdx])
} }
if matched == 0 { if matched == 0 {
return annotated, 0 return alignment{files: annotated}
} }
return annotated, total / float64(matched) a := alignment{
files: annotated,
titleFit: total / float64(matched),
timedPairs: timed,
aligned: matched,
}
if timed > 0 {
a.durationFit = durTotal / float64(timed)
}
return a
} }
// indexForPosition finds the expected track at a disc/track position. // indexForPosition finds the expected track at a disc/track position.
+224
View File
@@ -0,0 +1,224 @@
package download
import (
"context"
"errors"
"path/filepath"
"sync"
"testing"
"time"
)
// Soulseek politeness is per peer, not per daemon (#272).
func TestKeyedLockSerialisesOneKeyOnly(t *testing.T) {
t.Parallel()
var l keyedLock[string]
ctx := context.Background()
releaseA, err := l.acquire(ctx, "a")
if err != nil {
t.Fatalf("acquire a: %v", err)
}
// Another key is free while "a" is held.
releaseB, err := l.acquire(ctx, "b")
if err != nil {
t.Fatalf("acquire b: %v", err)
}
releaseB()
// The same key waits, and gives up with its context.
short, cancel := context.WithTimeout(ctx, 20*time.Millisecond)
defer cancel()
if _, err := l.acquire(short, "a"); !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("second acquire of a held key = %v, want the deadline", err)
}
releaseA()
releaseA() // Idempotent: a second call must not free someone else's hold.
if n := l.size(); n != 0 {
t.Errorf("%d keys left behind, want none once nobody holds or waits", n)
}
}
// grabEach runs one grab per candidate and returns a function that waits
// for all of them; grabAll's reasons for waiting apply.
func grabEach(t *testing.T, f managerFixture, cands []Candidate) func() {
t.Helper()
ctx := context.Background()
var wg sync.WaitGroup
for i, c := range cands {
dl := fourTrackDownload()
dl.ID = "dl-" + string(rune('a'+i))
if err := f.store.CreateDownload(ctx, dl); err != nil {
t.Fatalf("CreateDownload: %v", err)
}
wg.Add(1)
go func() {
defer wg.Done()
f.manager.grab(ctx, dl, c, nil, false)
}()
}
return func() {
done := make(chan struct{})
go func() {
wg.Wait()
close(done)
}()
select {
case <-done:
case <-time.After(5 * time.Second):
t.Error("transfers did not finish")
}
}
}
func slskdCandidates(p *FakeProvider, peers ...string) []Candidate {
out := make([]Candidate, 0, len(peers))
for i, peer := range peers {
c := p.Candidates[0]
c.ID = c.ID + "-" + itoa(i)
c.Kind = KindSlskd
c.ProviderID = 1
c.Origin = peer
out = append(out, c)
}
return out
}
// Three albums from one user are asked for one at a time, even though
// the daemon would allow three transfers.
func TestOnePeerIsAskedForOneThingAtATime(t *testing.T) {
t.Parallel()
f := newManagerFixture(t)
f.manager.SetMaxConcurrent(4)
p := fakeWithAlbum(1, "slskd", ".flac")
p.GrabGate = make(chan struct{})
f.manager.installProvider(Config{ID: 1, Kind: KindSlskd, Priority: 50}, p)
wait := grabEach(t, f, slskdCandidates(p, "alice", "alice", "alice"))
waitFor(t, func() bool { return p.GrabCallCount() >= 1 }, "no grab started")
time.Sleep(150 * time.Millisecond)
if got := p.MaxParallelGrabs(); got != 1 {
t.Errorf("%d simultaneous grabs from one peer, want 1", got)
}
close(p.GrabGate)
waitFor(t, func() bool { return p.GrabCallCount() == 3 }, "queued grabs never ran")
wait()
if n := f.manager.peerLocks.size(); n != 0 {
t.Errorf("%d peer locks left behind", n)
}
}
// Different users run at once, up to the daemon's cap — the point of
// the change: one slow peer no longer holds up every other.
func TestDifferentPeersRunTogether(t *testing.T) {
t.Parallel()
f := newManagerFixture(t)
f.manager.SetMaxConcurrent(8)
p := fakeWithAlbum(1, "slskd", ".flac")
p.GrabGate = make(chan struct{})
f.manager.installProvider(Config{ID: 1, Kind: KindSlskd, Priority: 50}, p)
wait := grabEach(t, f, slskdCandidates(p, "alice", "bob", "carol", "dave"))
waitFor(
t,
func() bool { return p.MaxParallelGrabs() >= kindConcurrency[KindSlskd] },
"different peers were serialised",
)
time.Sleep(100 * time.Millisecond)
if got := p.MaxParallelGrabs(); got != kindConcurrency[KindSlskd] {
t.Errorf("%d simultaneous grabs, want the daemon cap %d", got, kindConcurrency[KindSlskd])
}
close(p.GrabGate)
wait()
}
func TestSlskdLocalFolders(t *testing.T) {
t.Parallel()
s := &slskd{downloadsPath: "/dl"}
got := s.localFolders(Candidate{Files: []CandidateFile{
{Path: `\m\The Wall\CD2\01 Hey You.flac`},
{Path: `\m\The Wall\CD1\01 In The Flesh.flac`},
{Path: `\m\The Wall\CD1\02 The Thin Ice.flac`},
}})
want := []string{filepath.Join("/dl", "CD1"), filepath.Join("/dl", "CD2")}
if len(got) != len(want) || got[0] != want[0] || got[1] != want[1] {
t.Errorf("localFolders = %q, want %q", got, want)
}
}
// Two peers' "Greatest Hits" land in one slskd directory, so the second
// grab does not enqueue until the first has collected its files.
func TestSlskdSameFolderNameWaits(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
s, downloads := newStubSlskd(t, stub)
c := Candidate{
Payload: map[string]string{"username": "bob"},
Files: []CandidateFile{
{Path: `\music\Greatest Hits\01 Intro.flac`, Size: 1, IsAudio: true},
},
}
release, err := lockSlskdFolders(
context.Background(), []string{filepath.Join(downloads, "Greatest Hits")},
)
if err != nil {
t.Fatalf("lock: %v", err)
}
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
if _, err := s.Grab(ctx, c, t.TempDir(), nil); !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("Grab = %v, want it to wait on the held folder", err)
}
release()
stub.mu.Lock()
posted := stub.posted
stub.mu.Unlock()
if posted {
t.Error("enqueued transfers into a folder another grab held")
}
}
+8 -7
View File
@@ -236,17 +236,18 @@ func Register(d Descriptor, c Constructor) {
} }
// concurrencyField describes the per-provider transfer limit, with help // concurrencyField describes the per-provider transfer limit, with help
// text explaining why the default is what it is — a user who raises // text explaining what the number means where it means something
// slskd from 1 to 8 and gets themselves queued behind every other // unusual: on slskd it counts peers, since each peer is only ever asked
// Soulseek user deserves to have been warned. // for one folder at a time whatever it is set to.
func concurrencyField(k Kind) Field { func concurrencyField(k Kind) Field {
help := "Maximum simultaneous transfers from this client." help := "Maximum simultaneous transfers from this client."
if k == KindSlskd { if k == KindSlskd {
help = "Maximum simultaneous transfers. Soulseek peers serve " + help = "How many Soulseek users to download from at once. " +
"one file at a time and queue or ban clients that ask for " + "Each user is only ever asked for one album at a time, " +
"more, so 1 is both the polite setting and usually the " + "since peers queue or ban clients that ask for more; " +
"fastest." "this bounds how many different users are asked in " +
"parallel."
} }
return Field{ return Field{
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,130 @@
package download
import (
"context"
"os"
"path/filepath"
"slices"
"testing"
"time"
)
// TestSlskdLive runs the provider against a real slskd daemon. It is
// skipped unless YJ_SLSKD_URL, YJ_SLSKD_API_KEY and YJ_SLSKD_DOWNLOADS
// are set, and it downloads something only when YJ_SLSKD_GRAB=1 — then
// the smallest candidate the search returns, from whichever stranger
// is sharing it.
//
// Everything else here tests the provider against a stub written from
// reading slskd's source. This is where those readings are checked:
// the search options, the responses endpoint, the batch destination,
// the cancel.
//
// YJ_SLSKD_URL=http://localhost:5030 YJ_SLSKD_API_KEY=… \
// YJ_SLSKD_DOWNLOADS=/path/to/slskd/downloads YJ_SLSKD_GRAB=1 \
// go test -run TestSlskdLive -v ./backend/download/
func TestSlskdLive(t *testing.T) {
base, key, downloads := os.Getenv("YJ_SLSKD_URL"),
os.Getenv("YJ_SLSKD_API_KEY"), os.Getenv("YJ_SLSKD_DOWNLOADS")
if base == "" || key == "" || downloads == "" {
t.Skip(
"set YJ_SLSKD_URL, YJ_SLSKD_API_KEY and YJ_SLSKD_DOWNLOADS to run against a real slskd",
)
}
query := os.Getenv("YJ_SLSKD_QUERY")
if query == "" {
query = "Radiohead OK Computer"
}
p, err := newSlskd(
Config{
ID: 1, Kind: KindSlskd, Name: "live", Enabled: true,
Settings: map[string]string{"url": base, "downloadsPath": downloads},
},
func(string) (string, error) { return key, nil },
slogDiscard(),
)
if err != nil {
t.Fatalf("newSlskd: %v", err)
}
s, ok := p.(*slskd)
if !ok {
t.Fatalf("provider is %T", p)
}
ctx := context.Background()
if err := s.Check(ctx); err != nil {
t.Fatalf("Check: %v", err)
}
started := time.Now()
got, err := s.Search(ctx, Download{Query: query})
if err != nil {
t.Fatalf("Search: %v", err)
}
t.Logf(
"search %q: %d candidates in %s",
query,
len(got),
time.Since(started).Round(time.Millisecond),
)
if len(got) == 0 {
t.Fatal("no candidates; try a more common YJ_SLSKD_QUERY")
}
timed := 0
for _, c := range got {
for _, f := range c.Files {
if f.LengthMillis > 0 {
timed++
}
}
}
t.Logf("%d files carry a length", timed)
if os.Getenv("YJ_SLSKD_GRAB") != "1" {
return
}
smallest := slices.MinFunc(got, func(a, b Candidate) int {
return int(a.TotalSize - b.TotalSize)
})
t.Logf("grabbing %q from %s (%d files, %d bytes)",
smallest.Title, smallest.Origin, len(smallest.Files), smallest.TotalSize)
s.stallAfter = 3 * time.Minute
gctx, cancel := context.WithTimeout(ctx, 15*time.Minute)
defer cancel()
res, err := s.Grab(gctx, smallest, t.TempDir(), func(p Progress) {
t.Logf("%s: %d/%d bytes", p.Phase, p.Current, p.Total)
})
if err != nil {
// A stranger going offline is not a defect; what matters is
// that the transfers were cancelled, which slskd's UI shows.
t.Fatalf("Grab: %v", err)
}
t.Logf("batches: %v", s.batches.Load() == batchesSupported)
for _, f := range res.Files {
info, err := os.Stat(f)
if err != nil || info.Size() == 0 {
t.Errorf("collected %s is missing or empty: %v", f, err)
}
}
if entries, _ := os.ReadDir(filepath.Join(downloads, slskdDestRoot)); len(entries) != 0 {
t.Errorf("%d batch folders left in slskd's downloads", len(entries))
}
}
+499 -29
View File
@@ -7,7 +7,9 @@ import (
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"os" "os"
"path"
"path/filepath" "path/filepath"
"strconv"
"strings" "strings"
"sync" "sync"
"testing" "testing"
@@ -31,11 +33,45 @@ type slskdStub struct {
transfers [][]slskdTransfer transfers [][]slskdTransfer
pollCount int pollCount int
// before is what the downloads endpoint reports until something is
// enqueued: records slskd already held from earlier attempts.
before []slskdTransfer
// enqueued records what was requested for download. // enqueued records what was requested for download.
enqueued []map[string]any enqueued []map[string]any
posted bool
// paths records the escaped path of every transfers call, and
// cancelled the escaped request URI of every DELETE.
paths []string
cancelled []string
// unauthorized makes every call return 401. // unauthorized makes every call return 401.
unauthorized bool unauthorized bool
// searches records every search request body, and searchGets the
// request URI of every search GET.
searches []map[string]any
searchGets []string
// noResponsesEndpoint makes /searches/{id}/responses 404, as an
// older daemon would.
noResponsesEndpoint bool
// batches makes the daemon take batch downloads, as 0.26 does.
// Without it the batch endpoint answers 400, which is what an older
// daemon's per-user route does with a batch body. batchBodies
// records each batch, and delivered is written into its destination
// under downloads when it is enqueued, keyed by file base name.
batches bool
batchBodies []map[string]any
// positions is what the queue-position endpoint answers, in order;
// the last repeats. positionAsks counts the calls.
positions []int
positionAsks int
delivered map[string]string
downloads string
} }
func newSlskdStub(t *testing.T) *slskdStub { func newSlskdStub(t *testing.T) *slskdStub {
@@ -57,6 +93,16 @@ func newSlskdStub(t *testing.T) *slskdStub {
return return
} }
var body map[string]any
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
t.Errorf("decode search body: %v", err)
}
s.mu.Lock()
s.searches = append(s.searches, body)
s.mu.Unlock()
w.WriteHeader(http.StatusCreated) w.WriteHeader(http.StatusCreated)
}) })
@@ -73,8 +119,26 @@ func newSlskdStub(t *testing.T) *slskdStub {
s.mu.Lock() s.mu.Lock()
responses := s.responses responses := s.responses
noEndpoint := s.noResponsesEndpoint
s.searchGets = append(s.searchGets, r.URL.RequestURI())
s.mu.Unlock() s.mu.Unlock()
if strings.HasSuffix(r.URL.Path, "/responses") {
if noEndpoint {
w.WriteHeader(http.StatusNotFound)
return
}
writeJSON(t, w, responses)
return
}
if r.URL.Query().Get("includeResponses") != "true" {
responses = nil
}
writeJSON(t, w, slskdSearch{ writeJSON(t, w, slskdSearch{
ID: "search-1", ID: "search-1",
IsComplete: true, IsComplete: true,
@@ -87,7 +151,35 @@ func newSlskdStub(t *testing.T) *slskdStub {
return return
} }
if r.Method == http.MethodPost { s.mu.Lock()
s.paths = append(s.paths, r.URL.EscapedPath())
s.mu.Unlock()
if r.Method == http.MethodGet && strings.HasSuffix(r.URL.Path, "/position") {
s.mu.Lock()
idx := min(s.positionAsks, len(s.positions)-1)
s.positionAsks++
place := 0
if idx >= 0 {
place = s.positions[idx]
}
s.mu.Unlock()
writeJSON(t, w, place)
return
}
if r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/batches") {
s.enqueueBatch(t, w, r)
return
}
switch r.Method {
case http.MethodPost:
var body []map[string]any var body []map[string]any
if err := json.NewDecoder(r.Body).Decode(&body); err != nil { if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
@@ -96,15 +188,42 @@ func newSlskdStub(t *testing.T) *slskdStub {
s.mu.Lock() s.mu.Lock()
s.enqueued = body s.enqueued = body
s.posted = true
for _, file := range body {
name, _ := file["filename"].(string)
norm := strings.ReplaceAll(name, `\`, "/")
s.write(t, path.Base(path.Dir(norm)), path.Base(norm))
}
s.mu.Unlock() s.mu.Unlock()
w.WriteHeader(http.StatusCreated) w.WriteHeader(http.StatusCreated)
return
case http.MethodDelete:
s.mu.Lock()
s.cancelled = append(s.cancelled, r.URL.RequestURI())
s.mu.Unlock()
w.WriteHeader(http.StatusNoContent)
return return
} }
s.mu.Lock() s.mu.Lock()
if !s.posted {
before := s.before
s.mu.Unlock()
writeJSON(t, w, map[string]any{
"directories": []map[string]any{{"files": before}},
})
return
}
idx := s.pollCount idx := s.pollCount
if idx >= len(s.transfers) { if idx >= len(s.transfers) {
idx = len(s.transfers) - 1 idx = len(s.transfers) - 1
@@ -130,6 +249,89 @@ func newSlskdStub(t *testing.T) *slskdStub {
return s return s
} }
// enqueueBatch answers the batch endpoint.
func (s *slskdStub) enqueueBatch(t *testing.T, w http.ResponseWriter, r *http.Request) {
t.Helper()
s.mu.Lock()
defer s.mu.Unlock()
if !s.batches {
w.WriteHeader(http.StatusBadRequest)
return
}
var body map[string]any
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
t.Errorf("decode batch body: %v", err)
}
s.batchBodies = append(s.batchBodies, body)
s.posted = true
files, _ := body["files"].([]any)
options, _ := body["options"].(map[string]any)
dest, _ := options["destination"].(string)
for _, f := range files {
file, _ := f.(map[string]any)
s.enqueued = append(s.enqueued, file)
name, _ := file["filename"].(string)
base := path.Base(strings.ReplaceAll(name, `\`, "/"))
s.write(t, filepath.FromSlash(dest), base)
}
w.WriteHeader(http.StatusCreated)
}
// deliver names the files that arrive once enqueued.
func (s *slskdStub) deliver(names ...string) {
s.mu.Lock()
defer s.mu.Unlock()
if s.delivered == nil {
s.delivered = map[string]string{}
}
for _, n := range names {
s.delivered[n] = "audio"
}
}
// write puts a delivered file where slskd would: under dir in the
// downloads folder, renamed name_<ticks>.ext when the name is taken, as
// slskd's default Destination.Exists does. Callers hold s.mu.
func (s *slskdStub) write(t *testing.T, dir, base string) {
t.Helper()
content, ok := s.delivered[base]
if !ok {
return
}
full := filepath.Join(s.downloads, dir)
if err := os.MkdirAll(full, 0o750); err != nil {
t.Errorf("mkdir: %v", err)
}
target := filepath.Join(full, base)
if _, err := os.Stat(target); err == nil {
ext := filepath.Ext(base)
target = filepath.Join(
full,
strings.TrimSuffix(base, ext)+"_"+strconv.FormatInt(time.Now().UnixNano(), 10)+ext,
)
}
if err := os.WriteFile(target, []byte(content), 0o600); err != nil {
t.Errorf("write: %v", err)
}
}
// reject enforces API-key auth like the real daemon. // reject enforces API-key auth like the real daemon.
func (s *slskdStub) reject(w http.ResponseWriter, r *http.Request) bool { func (s *slskdStub) reject(w http.ResponseWriter, r *http.Request) bool {
s.mu.Lock() s.mu.Lock()
@@ -162,6 +364,10 @@ func newStubSlskd(t *testing.T, stub *slskdStub) (*slskd, string) {
downloads := t.TempDir() downloads := t.TempDir()
stub.mu.Lock()
stub.downloads = downloads
stub.mu.Unlock()
p, err := newSlskd( p, err := newSlskd(
Config{ Config{
ID: 1, ID: 1,
@@ -191,6 +397,13 @@ func newStubSlskd(t *testing.T, stub *slskdStub) (*slskd, string) {
s.searchWait = 200 * time.Millisecond s.searchWait = 200 * time.Millisecond
s.transferPoll = time.Millisecond s.transferPoll = time.Millisecond
// Long enough that no existing test trips them by accident; the
// tests about stalls and absences set their own.
s.stallAfter = time.Minute
s.absentGrace = time.Minute
s.positionPoll = time.Millisecond
s.queueCeiling = time.Hour
return s, downloads return s, downloads
} }
@@ -394,24 +607,9 @@ func TestSlskdGrabCollectsFromDownloadsFolder(t *testing.T) {
}, },
} }
s, downloads := newStubSlskd(t, stub) s, _ := newStubSlskd(t, stub)
// slskd writes into <downloads>/<folder>/<file>. stub.deliver("01 Airbag.flac", "02 Paranoid Android.flac")
folder := filepath.Join(downloads, "OK Computer")
if err := os.MkdirAll(folder, 0o750); err != nil {
t.Fatalf("mkdir: %v", err)
}
for _, name := range []string{
"01 Airbag.flac",
"02 Paranoid Android.flac",
} {
if err := os.WriteFile(
filepath.Join(folder, name), []byte("audio"), 0o600,
); err != nil {
t.Fatalf("write: %v", err)
}
}
c := Candidate{ c := Candidate{
ID: "slskd:peer:OK Computer", ID: "slskd:peer:OK Computer",
@@ -470,18 +668,9 @@ func TestSlskdGrabToleratesPartialFailure(t *testing.T) {
{Filename: `\s\Album\02 B.flac`, State: "Completed, Errored"}, {Filename: `\s\Album\02 B.flac`, State: "Completed, Errored"},
}} }}
s, downloads := newStubSlskd(t, stub) s, _ := newStubSlskd(t, stub)
folder := filepath.Join(downloads, "Album") stub.deliver("01 A.flac")
if err := os.MkdirAll(folder, 0o750); err != nil {
t.Fatalf("mkdir: %v", err)
}
if err := os.WriteFile(
filepath.Join(folder, "01 A.flac"), []byte("audio"), 0o600,
); err != nil {
t.Fatalf("write: %v", err)
}
c := Candidate{ c := Candidate{
Files: []CandidateFile{ Files: []CandidateFile{
@@ -565,3 +754,284 @@ func TestSlskdRequiresConfiguration(t *testing.T) {
}) })
} }
} }
// slskdAlbum is a two-file candidate from peer, whose arrived files
// the stub writes where slskd would once they are enqueued.
func slskdAlbum(t *testing.T, stub *slskdStub, peer string, arrived ...string) Candidate {
t.Helper()
stub.deliver(arrived...)
return Candidate{
Files: []CandidateFile{
{Path: `\s\Album\01 A.flac`, Size: 500, IsAudio: true},
{Path: `\s\Album\02 B.flac`, Size: 500, IsAudio: true},
},
TotalSize: 1000,
Payload: map[string]string{"username": peer},
}
}
// cancelledURIs returns what the stub was asked to cancel.
func (s *slskdStub) cancelledURIs() []string {
s.mu.Lock()
defer s.mu.Unlock()
return append([]string(nil), s.cancelled...)
}
// A peer that queues us and never sends a byte is given up on, and the
// queued transfers are cancelled in slskd rather than left to start
// hours later for a request nobody is waiting on.
func TestSlskdGrabGivesUpOnAStalledPeer(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
stub.transfers = [][]slskdTransfer{{
{ID: "t1", Filename: `\s\Album\01 A.flac`, State: "Queued, Remotely"},
{ID: "t2", Filename: `\s\Album\02 B.flac`, State: "Queued, Remotely"},
}}
s, _ := newStubSlskd(t, stub)
s.stallAfter = 30 * time.Millisecond
_, err := s.Grab(
context.Background(), slskdAlbum(t, stub, "peer"), t.TempDir(), nil,
)
if !errors.Is(err, ErrSlskdTimeout) {
t.Fatalf("error = %v, want ErrSlskdTimeout", err)
}
got := stub.cancelledURIs()
if len(got) != 2 {
t.Fatalf("cancelled %v, want both queued transfers", got)
}
for _, uri := range got {
if !strings.HasSuffix(uri, "?remove=true") {
t.Errorf("cancel %s does not remove the record", uri)
}
}
}
// A folder that stalls on its last track goes forward with what
// arrived, the same as one whose last track failed; the importer's
// completeness check decides whether that is enough.
func TestSlskdGrabKeepsWhatArrivedBeforeAStall(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
stub.transfers = [][]slskdTransfer{{
{
ID: "t1", Filename: `\s\Album\01 A.flac`,
State: "Completed, Succeeded", BytesTransferred: 500,
},
{ID: "t2", Filename: `\s\Album\02 B.flac`, State: "Queued, Remotely"},
}}
s, _ := newStubSlskd(t, stub)
s.stallAfter = 30 * time.Millisecond
got, err := s.Grab(
context.Background(),
slskdAlbum(t, stub, "peer", "01 A.flac"),
t.TempDir(), nil,
)
if err != nil {
t.Fatalf("Grab: %v", err)
}
if len(got.Files) != 1 {
t.Errorf("collected %d files, want the 1 that arrived", len(got.Files))
}
if cancelled := stub.cancelledURIs(); len(cancelled) != 1 ||
!strings.Contains(cancelled[0], "/t2") {
t.Errorf("cancelled %v, want only the stalled t2", cancelled)
}
}
// Progress is what holds the stall timer off. A transfer that keeps
// moving bytes is never abandoned, however long it takes.
func TestSlskdGrabWaitsOnATransferThatIsMoving(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
for b := int64(1); b <= 100; b++ {
stub.transfers = append(stub.transfers, []slskdTransfer{
{ID: "t1", Filename: `\s\Album\01 A.flac`, State: "InProgress", BytesTransferred: b},
{ID: "t2", Filename: `\s\Album\02 B.flac`, State: "Queued, Remotely"},
})
}
stub.transfers = append(stub.transfers, []slskdTransfer{
{
ID: "t1",
Filename: `\s\Album\01 A.flac`,
State: "Completed, Succeeded",
BytesTransferred: 500,
},
{
ID: "t2",
Filename: `\s\Album\02 B.flac`,
State: "Completed, Succeeded",
BytesTransferred: 500,
},
})
s, _ := newStubSlskd(t, stub)
// A hundred polls take several times the stall window; each one
// moves a byte. The window is kept well above one poll so a
// descheduled test runner does not read as a stall.
s.transferPoll = 5 * time.Millisecond
s.stallAfter = 150 * time.Millisecond
got, err := s.Grab(
context.Background(),
slskdAlbum(t, stub, "peer", "01 A.flac", "02 B.flac"),
t.TempDir(), nil,
)
if err != nil {
t.Fatalf("Grab: %v", err)
}
if len(got.Files) != 2 {
t.Errorf("collected %d files, want 2", len(got.Files))
}
}
// A file slskd never lists was refused at enqueue and will never reach
// a terminal state. It counts as failed once the grace period is up,
// rather than being waited on until the six-hour ceiling.
func TestSlskdGrabCountsAnUnlistedFileAsFailed(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
stub.transfers = [][]slskdTransfer{{
{
ID: "t1", Filename: `\s\Album\01 A.flac`,
State: "Completed, Succeeded", BytesTransferred: 500,
},
}}
s, _ := newStubSlskd(t, stub)
s.absentGrace = 20 * time.Millisecond
got, err := s.Grab(
context.Background(),
slskdAlbum(t, stub, "peer", "01 A.flac"),
t.TempDir(), nil,
)
if err != nil {
t.Fatalf("Grab: %v", err)
}
if len(got.Files) != 1 {
t.Errorf("collected %d files, want 1", len(got.Files))
}
}
// A finished record left by an earlier attempt at the same file is not
// this attempt's answer. Without the snapshot it would fail the grab on
// the first poll, before the new transfer had started.
func TestSlskdGrabIgnoresAnEarlierAttemptsRecord(t *testing.T) {
t.Parallel()
stale := slskdTransfer{
ID: "old", Filename: `\s\Album\01 A.flac`, State: "Completed, Errored",
}
stub := newSlskdStub(t)
stub.before = []slskdTransfer{stale}
stub.transfers = [][]slskdTransfer{
{stale},
{
stale,
{
ID: "new", Filename: `\s\Album\01 A.flac`,
State: "Completed, Succeeded", BytesTransferred: 500,
},
},
}
s, _ := newStubSlskd(t, stub)
c := slskdAlbum(t, stub, "peer", "01 A.flac")
c.Files = c.Files[:1]
got, err := s.Grab(context.Background(), c, t.TempDir(), nil)
if err != nil {
t.Fatalf("Grab: %v", err)
}
if len(got.Files) != 1 {
t.Errorf("collected %d files, want 1", len(got.Files))
}
}
// Cancelling the download cancels the transfer in slskd too. The
// cleanup must not inherit the cancelled context, or it is never sent.
func TestSlskdGrabCancelsTransfersWhenTheCallerGivesUp(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
stub.transfers = [][]slskdTransfer{{
{ID: "t1", Filename: `\s\Album\01 A.flac`, State: "InProgress", BytesTransferred: 10},
{ID: "t2", Filename: `\s\Album\02 B.flac`, State: "Queued, Remotely"},
}}
s, _ := newStubSlskd(t, stub)
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
defer cancel()
_, err := s.Grab(ctx, slskdAlbum(t, stub, "peer"), t.TempDir(), nil)
if !errors.Is(err, ErrSlskdTimeout) {
t.Fatalf("error = %v, want ErrSlskdTimeout", err)
}
if got := stub.cancelledURIs(); len(got) != 2 {
t.Errorf("cancelled %v, want both live transfers", got)
}
}
// Soulseek usernames carry spaces and punctuation; spliced raw into the
// path, a name with a slash addresses a different endpoint entirely.
func TestSlskdEscapesTheUsername(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
stub.transfers = [][]slskdTransfer{{
{
ID: "t1", Filename: `\s\Album\01 A.flac`,
State: "Completed, Succeeded", BytesTransferred: 500,
},
{
ID: "t2", Filename: `\s\Album\02 B.flac`,
State: "Completed, Succeeded", BytesTransferred: 500,
},
}}
s, _ := newStubSlskd(t, stub)
if _, err := s.Grab(
context.Background(),
slskdAlbum(t, stub, "dj a/b", "01 A.flac", "02 B.flac"),
t.TempDir(), nil,
); err != nil {
t.Fatalf("Grab: %v", err)
}
stub.mu.Lock()
paths := append([]string(nil), stub.paths...)
stub.mu.Unlock()
for _, p := range paths {
// The batch endpoint carries the name in its body.
if p != "/api/v0/transfers/downloads/dj%20a%2Fb" &&
p != "/api/v0/transfers/downloads/batches" {
t.Errorf("transfers call went to %s", p)
}
}
}
+104 -18
View File
@@ -35,6 +35,15 @@ const (
weightArtistFit = 0.12 weightArtistFit = 0.12
) )
// Match sub-weights when the candidate's durations are known. Duration
// takes its weight from title fit, the signal it corroborates: a title
// says which song a file claims to be, a length says whether it is that
// recording — the right edit, the whole file, not the live take.
const (
timedWeightTitleFit = 0.25
timedWeightDurationFit = 0.15
)
// Quality sub-weights. Each set sums to 1.0. // Quality sub-weights. Each set sums to 1.0.
// //
// There are two of them because a stated preference changes what the // There are two of them because a stated preference changes what the
@@ -319,13 +328,13 @@ func Score(dl Download, c Candidate, priority int, prefs AutoDownloadPrefs) Cand
audio := c.AudioFiles() audio := c.AudioFiles()
matched, titleFit := matchFiles(audio, dl.Expected) a := alignFiles(audio, dl.Expected)
// Write the alignment back so the picker can show which file maps // Write the alignment back so the picker can show which file maps
// to which track. // to which track.
c.Files = mergeMatched(c.Files, matched) c.Files = mergeMatched(c.Files, a.files)
c.Match = scoreMatch(dl, c, audio, titleFit) c.Match = scoreMatch(dl, c, audio, a)
c.Quality = scoreQuality( c.Quality = scoreQuality(
c, audio, priority, prefs, dl.runtimeMillis(), c, audio, priority, prefs, dl.runtimeMillis(),
) )
@@ -340,14 +349,21 @@ func scoreMatch(
dl Download, dl Download,
c Candidate, c Candidate,
audio []CandidateFile, audio []CandidateFile,
titleFit float64, a alignment,
) MatchScore { ) MatchScore {
m := MatchScore{ m := MatchScore{
Anchored: dl.Anchored(), Anchored: dl.Anchored(),
TitleFit: titleFit, TitleFit: a.titleFit,
DurationFit: a.durationFit,
// Durations count once at least half the aligned pairs state
// one; a single timed pair would be a coin toss carrying 15%.
DurationKnown: a.timedPairs > 0 && a.timedPairs*2 >= a.aligned,
} }
m.Completeness = completeness(len(audio), len(dl.Expected)) m.Completeness = completeness(
alignedCount(c.Files), len(audio), len(dl.Expected),
)
// The candidate's own title, and the folder its files sit in, are // The candidate's own title, and the folder its files sit in, are
// two independent guesses at the album name. Take the better one: // two independent guesses at the album name. Take the better one:
@@ -367,9 +383,16 @@ func scoreMatch(
// With no expected tracklist there is no title signal at all, so // With no expected tracklist there is no title signal at all, so
// redistribute its weight onto the album/artist evidence rather // redistribute its weight onto the album/artist evidence rather
// than scoring every free-text result as half-wrong. // than scoring every free-text result as half-wrong.
if len(dl.Expected) == 0 { switch {
case len(dl.Expected) == 0:
m.Overall = 0.55*m.AlbumFit + 0.45*m.ArtistFit m.Overall = 0.55*m.AlbumFit + 0.45*m.ArtistFit
} else { case m.DurationKnown:
m.Overall = timedWeightTitleFit*m.TitleFit +
timedWeightDurationFit*m.DurationFit +
weightCompleteness*m.Completeness +
weightAlbumFit*m.AlbumFit +
weightArtistFit*m.ArtistFit
default:
m.Overall = weightTitleFit*m.TitleFit + m.Overall = weightTitleFit*m.TitleFit +
weightCompleteness*m.Completeness + weightCompleteness*m.Completeness +
weightAlbumFit*m.AlbumFit + weightAlbumFit*m.AlbumFit +
@@ -415,30 +438,54 @@ func artistFit(want string, c Candidate) float64 {
return best return best
} }
// completeness scores audio file count against the expected track // completeness scores how much of the expected tracklist a candidate
// count. Extra files are penalized far more gently than missing ones: // covers. Extra files are penalized far more gently than missing ones:
// a folder with bonus tracks or a stray intro is still the album, while // a folder with bonus tracks or a stray intro is still the album, while
// a folder missing half the tracks is not. // a folder missing half the tracks is not.
func completeness(got, want int) float64 { //
// **Coverage is counted in aligned tracks, not in files.** It used to
// be the audio file count, so any ten files scored full marks against
// a ten-track album whether or not they were its tracks — and since
// title fit is the mean over the files that *did* align, a folder where
// three titles matched read as a near-perfect candidate on both counts.
// `aligned` is how many files matchFiles assigned to an expected track;
// `audio` still sets the penalty for extras, because a folder of thirty
// files holding the ten wanted is a worse copy than one holding ten.
func completeness(aligned, audio, want int) float64 {
if want == 0 { if want == 0 {
if got > 0 { if audio > 0 {
return 0.5 return 0.5
} }
return 0 return 0
} }
if got == 0 { if aligned == 0 {
return 0 return 0
} }
if got >= want { cover := float64(min(aligned, want)) / float64(want)
extra := float64(got-want) / float64(want)
return math.Max(0.75, 1.0-0.25*extra) if audio > want {
extra := float64(audio-want) / float64(want)
cover *= math.Max(0.75, 1.0-0.25*extra)
} }
return float64(got) / float64(want) return cover
}
// alignedCount is how many audio files were assigned to an expected
// track.
func alignedCount(files []CandidateFile) int {
n := 0
for _, f := range files {
if f.IsAudio && f.MatchedTo != 0 {
n++
}
}
return n
} }
// scoreQuality answers whether this is a good copy. // scoreQuality answers whether this is a good copy.
@@ -735,6 +782,45 @@ func AutoPickVeto(
return "" return ""
} }
// autoAcceptable reports whether auto-pick may take this one candidate
// without asking: the request is anchored to a tracklist, and the
// candidate is inside the user's guardrails and clears the match and
// quality bars. It is AutoPickVeto's test applied to a single
// candidate, which is what falling back to a second choice needs.
func autoAcceptable(dl Download, c Candidate, prefs AutoDownloadPrefs) bool {
return dl.Anchored() &&
len(dl.Expected) > 0 &&
prefs.eligible(c, dl.runtimeMillis()) &&
c.Match.Overall >= minMatch &&
c.Quality.Overall >= minQuality
}
// autoPick returns the candidate auto-pick takes: the best-ranked one
// it may take at all.
//
// That is not `ranked[0]`. AutoPickVeto judges the best candidate
// *inside* the guardrails, so when the overall best is outside them —
// over the size ceiling, say — the veto passes on the strength of the
// second, and grabbing the first would download exactly the copy the
// user said not to take unattended.
func autoPick(
dl Download,
ranked []Candidate,
prefs AutoDownloadPrefs,
) (Candidate, bool) {
if AutoPickVeto(dl, ranked, prefs) != "" {
return Candidate{}, false
}
for _, c := range ranked {
if autoAcceptable(dl, c, prefs) {
return c, true
}
}
return Candidate{}, false
}
// mergeMatched copies MatchedTo assignments from the audio-only slice // mergeMatched copies MatchedTo assignments from the audio-only slice
// back onto the full file list. // back onto the full file list.
func mergeMatched(all, matched []CandidateFile) []CandidateFile { func mergeMatched(all, matched []CandidateFile) []CandidateFile {
+14 -10
View File
@@ -282,28 +282,32 @@ func TestCompleteness(t *testing.T) {
tests := []struct { tests := []struct {
name string name string
got int aligned int
audio int
want int want int
minScore float64 minScore float64
maxScore float64 maxScore float64
}{ }{
{"exact", 10, 10, 1.0, 1.0}, {"exact", 10, 10, 10, 1.0, 1.0},
{"half missing", 5, 10, 0.49, 0.51}, {"half missing", 5, 5, 10, 0.49, 0.51},
{"one bonus track", 11, 10, 0.95, 1.0}, {"one bonus track", 10, 11, 10, 0.95, 1.0},
{"double", 20, 10, 0.74, 0.76}, {"double", 10, 20, 10, 0.74, 0.76},
{"nothing", 0, 10, 0, 0}, {"nothing", 0, 0, 10, 0, 0},
{"no expectation", 5, 0, 0.5, 0.5}, {"no expectation", 0, 5, 0, 0.5, 0.5},
// Ten files are not ten tracks: three that align are three.
{"right count, wrong tracks", 3, 10, 10, 0.29, 0.31},
{"files that align to nothing", 0, 10, 10, 0, 0},
} }
for _, tt := range tests { for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
t.Parallel() t.Parallel()
got := completeness(tt.got, tt.want) got := completeness(tt.aligned, tt.audio, tt.want)
if got < tt.minScore || got > tt.maxScore { if got < tt.minScore || got > tt.maxScore {
t.Errorf( t.Errorf(
"completeness(%d, %d) = %f, want in [%f, %f]", "completeness(%d, %d, %d) = %f, want in [%f, %f]",
tt.got, tt.want, got, tt.minScore, tt.maxScore, tt.aligned, tt.audio, tt.want, got, tt.minScore, tt.maxScore,
) )
} }
}) })
+238
View File
@@ -0,0 +1,238 @@
package download
import (
"context"
"strings"
"testing"
)
// What Soulseek is asked, how, and what is kept from the answer (#271).
func TestSlskdQueries(t *testing.T) {
t.Parallel()
cases := []struct {
name string
dl Download
want []string
}{
{
name: "a plain request is searched once",
dl: Download{Artist: "Radiohead", Album: "OK Computer"},
want: []string{"Radiohead OK Computer"},
},
{
name: "an edition qualifier gets a second query without it",
dl: Download{Artist: "Radiohead", Album: "OK Computer (Collector's Edition)"},
want: []string{
"Radiohead OK Computer (Collector's Edition)",
"Radiohead OK Computer",
},
},
{
name: "a trailing remaster note",
dl: Download{Artist: "Pink Floyd", Album: "Animals - 2018 Remaster"},
want: []string{
"Pink Floyd Animals - 2018 Remaster",
"Pink Floyd Animals",
},
},
{
name: "a leading dash would be an exclusion",
dl: Download{Artist: "Mocky", Album: "-ism"},
want: []string{"Mocky -ism", "Mocky ism"},
},
{
name: "a compilation is not searched by its placeholder artist",
dl: Download{Artist: "Various Artists", Album: "Pulp Fiction"},
want: []string{"Various Artists Pulp Fiction", "Pulp Fiction"},
},
{
name: "what the user typed is searched as written",
dl: Download{Query: "ok computer (deluxe)", Album: "OK Computer (Deluxe)"},
want: []string{"ok computer (deluxe)"},
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
got := slskdQueries(tc.dl)
if strings.Join(got, "|") != strings.Join(tc.want, "|") {
t.Errorf("slskdQueries = %q, want %q", got, tc.want)
}
})
}
}
// Both queries run, the options are stated rather than left to the
// daemon's defaults, and a folder both queries found is one candidate.
func TestSlskdSearchRunsBothQueriesAndMerges(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
stub.responses = []slskdResponse{{
Username: "peer",
Files: []slskdFile{
{Filename: `\m\Radiohead - OK Computer\01 Airbag.flac`, Size: 1, Length: 284},
{Filename: `\m\Radiohead - OK Computer\02 Paranoid Android.flac`, Size: 1, Length: 383},
},
}}
s, _ := newStubSlskd(t, stub)
got, err := s.Search(context.Background(), Download{
ReleaseMBID: "rel", Artist: "Radiohead", Album: "OK Computer (Deluxe Edition)",
})
if err != nil {
t.Fatalf("Search: %v", err)
}
if len(got) != 1 {
t.Fatalf("got %d candidates, want the one folder once", len(got))
}
if got[0].Files[0].LengthMillis != 284_000 {
t.Errorf("length = %d ms, want 284000 from slskd's seconds", got[0].Files[0].LengthMillis)
}
stub.mu.Lock()
searches := append([]map[string]any(nil), stub.searches...)
gets := append([]string(nil), stub.searchGets...)
stub.mu.Unlock()
if len(searches) != 2 {
t.Fatalf("ran %d searches, want 2", len(searches))
}
for _, body := range searches {
for _, key := range []string{
"searchTimeout", "responseLimit", "fileLimit",
"minimumResponseFileCount", "maximumPeerQueueLength",
} {
if _, ok := body[key]; !ok {
t.Errorf("search %q does not state %s", body["searchText"], key)
}
}
}
// The responses are fetched once at the end, not with every poll.
for _, uri := range gets {
if strings.Contains(uri, "includeResponses") {
t.Errorf("poll %s asked for every response", uri)
}
}
}
// A daemon without the responses endpoint still returns results.
func TestSlskdSearchFallsBackForAnOlderDaemon(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
stub.noResponsesEndpoint = true
stub.responses = []slskdResponse{{
Username: "peer",
Files: []slskdFile{
{Filename: `\m\Album\01 A.flac`, Size: 1},
{Filename: `\m\Album\02 B.flac`, Size: 1},
},
}}
s, _ := newStubSlskd(t, stub)
got, err := s.Search(context.Background(), Download{Query: "album"})
if err != nil {
t.Fatalf("Search: %v", err)
}
if len(got) != 1 {
t.Errorf("got %d candidates, want 1 through the fallback", len(got))
}
}
func TestDurationAgreement(t *testing.T) {
t.Parallel()
cases := []struct {
got, want int64
score float64
}{
{300_000, 300_000, 1},
{301_500, 300_000, 1}, // a second of silence
{300_000, 316_500, 0.5},
{300_000, 345_000, 0}, // a different edit
}
for _, tc := range cases {
if got := durationAgreement(tc.got, tc.want); got < tc.score-0.01 || got > tc.score+0.01 {
t.Errorf("durationAgreement(%d, %d) = %f, want %f", tc.got, tc.want, got, tc.score)
}
}
}
// Two folders with the same track names are told apart by their
// lengths: one is the album, the other a live record of the same songs.
func TestDurationsSeparateTheRightRecording(t *testing.T) {
t.Parallel()
dl := okComputer()
timed := func(id string, lengths ...int64) Candidate {
c := candidateFor(id, allTitles(), ".flac", 30_000_000)
for i := range c.Files {
c.Files[i].LengthMillis = lengths[i]
}
return c
}
studio := timed("studio", trackMillis, trackMillis+1_000, trackMillis, trackMillis-500)
live := timed(
"live",
trackMillis+60_000,
trackMillis+75_000,
trackMillis+50_000,
trackMillis+90_000,
)
ranked := Rank(dl, []Candidate{live, studio}, nil, AutoDownloadPrefs{})
if ranked[0].ID != "studio" {
t.Fatalf("winner = %s, want the recording whose lengths match", ranked[0].ID)
}
if !ranked[0].Match.DurationKnown || ranked[0].Match.DurationFit < 0.99 {
t.Errorf(
"studio duration fit = %f known=%v",
ranked[0].Match.DurationFit,
ranked[0].Match.DurationKnown,
)
}
if ranked[1].Match.DurationFit != 0 {
t.Errorf("live duration fit = %f, want 0", ranked[1].Match.DurationFit)
}
}
// Without lengths the score is exactly what it was before durations
// were read, so a provider that reports none is not penalised.
func TestUnknownDurationsLeaveTheScoreAlone(t *testing.T) {
t.Parallel()
dl := okComputer()
c := Score(dl, candidateFor("c", allTitles(), ".flac", 30_000_000), 50, AutoDownloadPrefs{})
if c.Match.DurationKnown {
t.Fatal("no file states a length, yet durations are known")
}
want := weightTitleFit*c.Match.TitleFit +
weightCompleteness*c.Match.Completeness +
weightAlbumFit*c.Match.AlbumFit +
weightArtistFit*c.Match.ArtistFit
if c.Match.Overall != want {
t.Errorf("match = %f, want the untimed formula's %f", c.Match.Overall, want)
}
}
+328
View File
@@ -0,0 +1,328 @@
package download
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// Where slskd writes a grab's files, and how collect finds them (#274).
// slskd reads searchTimeout in whole seconds, from the last response.
func TestSlskdSearchTimeoutIsInSeconds(t *testing.T) {
t.Parallel()
cases := []struct {
wait time.Duration
want int
}{
{20 * time.Second, 18},
{200 * time.Millisecond, slskdMinSearchTimeout},
}
for _, tc := range cases {
s := &slskd{searchWait: tc.wait}
got, ok := s.searchRequest("id", "text", 2)["searchTimeout"].(int)
if !ok || got != tc.want {
t.Errorf("wait %s: searchTimeout = %v, want %d seconds", tc.wait, got, tc.want)
}
}
}
func TestIsRenamedCopy(t *testing.T) {
t.Parallel()
cases := map[string]bool{
"01 A_638912345678901234.flac": true,
"01 A.flac": false,
"01 A_.flac": false,
"01 A_v2.flac": false,
"01 A_123.mp3": false,
"01 AB_123.flac": false,
}
for name, want := range cases {
if got := isRenamedCopy(name, "01 A", ".flac"); got != want {
t.Errorf("isRenamedCopy(%q) = %v, want %v", name, got, want)
}
}
}
func succeeded(names ...string) [][]slskdTransfer {
out := make([]slskdTransfer, 0, len(names))
for i, n := range names {
out = append(out, slskdTransfer{
ID: "t" + itoa(i),
Filename: n,
State: "Completed, Succeeded",
BytesTransferred: 500,
})
}
return [][]slskdTransfer{out}
}
// On a daemon with batches, each grab writes into a folder of its own,
// collect reads from exactly there, and the folder is gone afterwards.
func TestSlskdBatchGrabUsesItsOwnFolder(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
stub.batches = true
stub.transfers = succeeded(`\s\Album\01 A.flac`, `\s\Album\02 B.flac`)
s, downloads := newStubSlskd(t, stub)
// A same-named file in the folder a per-user enqueue would use is
// someone else's, and must not be touched.
other := filepath.Join(downloads, "Album", "01 A.flac")
if err := os.MkdirAll(filepath.Dir(other), 0o750); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(other, []byte("the user's"), 0o600); err != nil {
t.Fatal(err)
}
dst := t.TempDir()
got, err := s.Grab(
context.Background(), slskdAlbum(t, stub, "peer", "01 A.flac", "02 B.flac"), dst, nil,
)
if err != nil {
t.Fatalf("Grab: %v", err)
}
if len(got.Files) != 2 {
t.Fatalf("collected %d files, want 2", len(got.Files))
}
stub.mu.Lock()
bodies := append([]map[string]any(nil), stub.batchBodies...)
stub.mu.Unlock()
if len(bodies) != 1 {
t.Fatalf("%d batches, want 1", len(bodies))
}
if bodies[0]["username"] != "peer" {
t.Errorf("batch username = %v", bodies[0]["username"])
}
dest, _ := bodies[0]["options"].(map[string]any)["destination"].(string)
if !strings.HasPrefix(dest, slskdDestRoot+"/") {
t.Errorf("destination %q is not under %s", dest, slskdDestRoot)
}
if _, err := os.Stat(filepath.Join(downloads, filepath.FromSlash(dest))); !os.IsNotExist(err) {
t.Errorf("batch folder left behind: %v", err)
}
if data, _ := os.ReadFile(other); string(data) != "the user's" {
t.Errorf("the user's own file was taken or changed: %q", data)
}
}
// A batch's files land flat in its destination, so a two-disc rip is
// two batches, one per disc, or disc 2's "01" is renamed out of the way
// of disc 1's.
func TestSlskdBatchSplitsDiscs(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
stub.batches = true
stub.transfers = succeeded(`\s\Wall\CD1\01 In.flac`, `\s\Wall\CD2\01 Hey You.flac`)
stub.deliver("01 In.flac", "01 Hey You.flac")
s, _ := newStubSlskd(t, stub)
dst := t.TempDir()
got, err := s.Grab(context.Background(), Candidate{
Files: []CandidateFile{
{Path: `\s\Wall\CD1\01 In.flac`, Size: 500, IsAudio: true},
{Path: `\s\Wall\CD2\01 Hey You.flac`, Size: 500, IsAudio: true},
},
Payload: map[string]string{"username": "peer"},
}, dst, nil)
if err != nil {
t.Fatalf("Grab: %v", err)
}
stub.mu.Lock()
n := len(stub.batchBodies)
stub.mu.Unlock()
if n != 2 {
t.Errorf("%d batches, want one per disc", n)
}
for _, want := range []string{
filepath.Join(dst, "CD1", "01 In.flac"),
filepath.Join(dst, "CD2", "01 Hey You.flac"),
} {
found := false
for _, f := range got.Files {
found = found || f == want
}
if !found {
t.Errorf("%s not collected; got %q", want, got.Files)
}
}
}
// An abandoned batch grab leaves nothing in slskd's folder either.
func TestSlskdBatchFailureDiscardsItsFolder(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
stub.batches = true
stub.transfers = [][]slskdTransfer{{
{ID: "t0", Filename: `\s\Album\01 A.flac`, State: "Completed, Errored"},
{ID: "t1", Filename: `\s\Album\02 B.flac`, State: "Completed, Errored"},
}}
s, downloads := newStubSlskd(t, stub)
// The stub delivers the file, as a partial slskd left behind would.
if _, err := s.Grab(
context.Background(), slskdAlbum(t, stub, "peer", "01 A.flac"), t.TempDir(), nil,
); err == nil {
t.Fatal("Grab succeeded with every transfer failed")
}
entries, _ := os.ReadDir(filepath.Join(downloads, slskdDestRoot))
if len(entries) != 0 {
t.Errorf("%d batch folders left behind", len(entries))
}
}
// An older daemon answers the batch endpoint with 400; the grab falls
// back to the per-user enqueue, and later grabs do not ask again.
func TestSlskdFallsBackWithoutBatches(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
stub.transfers = succeeded(`\s\Album\01 A.flac`, `\s\Album\02 B.flac`)
s, _ := newStubSlskd(t, stub)
for range 2 {
stub.mu.Lock()
stub.posted = false
stub.pollCount = 0
stub.mu.Unlock()
if _, err := s.Grab(
context.Background(), slskdAlbum(t, stub, "peer", "01 A.flac"), t.TempDir(), nil,
); err != nil {
t.Fatalf("Grab: %v", err)
}
}
stub.mu.Lock()
paths := append([]string(nil), stub.paths...)
stub.mu.Unlock()
batchCalls := 0
for _, p := range paths {
if strings.HasSuffix(p, "/batches") {
batchCalls++
}
}
if batchCalls != 1 {
t.Errorf("asked for a batch %d times, want once", batchCalls)
}
}
// Without batches, a file of the same name already in slskd's folder is
// not this grab's: slskd wrote ours beside it as name_<ticks>.ext, and
// that is the one collected.
func TestSlskdCollectsTheRenamedCopyNotTheOldFile(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
stub.transfers = succeeded(`\s\Album\01 A.flac`, `\s\Album\02 B.flac`)
s, downloads := newStubSlskd(t, stub)
old := filepath.Join(downloads, "Album", "01 A.flac")
if err := os.MkdirAll(filepath.Dir(old), 0o750); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(old, []byte("left by an earlier attempt"), 0o600); err != nil {
t.Fatal(err)
}
dst := t.TempDir()
got, err := s.Grab(
context.Background(), slskdAlbum(t, stub, "peer", "01 A.flac", "02 B.flac"), dst, nil,
)
if err != nil {
t.Fatalf("Grab: %v", err)
}
if len(got.Files) != 2 {
t.Fatalf("collected %d files, want 2", len(got.Files))
}
data, err := os.ReadFile(filepath.Join(dst, "01 A.flac"))
if err != nil || string(data) != "audio" {
t.Errorf("collected %q, want this grab's file", data)
}
if data, _ := os.ReadFile(old); string(data) != "left by an earlier attempt" {
t.Error("the file that was already there was moved")
}
}
// And when this grab's copy never arrived, the old one is not taken in
// its place.
func TestSlskdDoesNotCollectAFileThatWasAlreadyThere(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
stub.transfers = [][]slskdTransfer{
{
{ID: "t0", Filename: `\s\Album\01 A.flac`, State: "Completed, Errored"},
{
ID: "t1",
Filename: `\s\Album\02 B.flac`,
State: "Completed, Succeeded",
BytesTransferred: 500,
},
},
}
s, downloads := newStubSlskd(t, stub)
old := filepath.Join(downloads, "Album", "01 A.flac")
if err := os.MkdirAll(filepath.Dir(old), 0o750); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(old, []byte("stale"), 0o600); err != nil {
t.Fatal(err)
}
got, err := s.Grab(
context.Background(), slskdAlbum(t, stub, "peer", "02 B.flac"), t.TempDir(), nil,
)
if err != nil {
t.Fatalf("Grab: %v", err)
}
if len(got.Files) != 1 || filepath.Base(got.Files[0]) != "02 B.flac" {
t.Errorf("collected %q, want only 02 B.flac", got.Files)
}
}
+132
View File
@@ -0,0 +1,132 @@
package download
import (
"context"
"errors"
"strings"
"testing"
"time"
)
// A peer that has queued us is judged by where we are in its queue, not
// only by a timer (#275).
func queuedThen(polls int, final string) [][]slskdTransfer {
queued := []slskdTransfer{
{ID: "t0", Filename: `\s\Album\01 A.flac`, State: "Queued, Remotely"},
{ID: "t1", Filename: `\s\Album\02 B.flac`, State: "Queued, Remotely"},
}
out := make([][]slskdTransfer, 0, polls+1)
for range polls {
out = append(out, queued)
}
return append(out, []slskdTransfer{
{ID: "t0", Filename: `\s\Album\01 A.flac`, State: final, BytesTransferred: 500},
{ID: "t1", Filename: `\s\Album\02 B.flac`, State: final, BytesTransferred: 500},
})
}
func descending(from int) []int {
out := make([]int, 0, from)
for p := from; p >= 1; p-- {
out = append(out, p)
}
return out
}
// Two readings far back in the queue give the peer up at once, rather
// than after the stall timer.
func TestSlskdGivesUpOnALongQueue(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
stub.transfers = queuedThen(100_000, "Completed, Succeeded")
stub.positions = []int{400}
s, _ := newStubSlskd(t, stub)
s.stallAfter = time.Hour
started := time.Now()
_, err := s.Grab(context.Background(), slskdAlbum(t, stub, "peer"), t.TempDir(), nil)
if !errors.Is(err, ErrSlskdTimeout) || !strings.Contains(err.Error(), "position 400") {
t.Fatalf("Grab = %v, want a queue-position give-up", err)
}
if time.Since(started) > 5*time.Second {
t.Error("the give-up waited on something other than the position")
}
if len(stub.cancelledURIs()) == 0 {
t.Error("the queued transfers were not cancelled")
}
}
// One far reading is not enough: slskd says the figure can be wildly
// wrong.
func TestSlskdOneBadPositionIsNotEnough(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
stub.transfers = queuedThen(20, "Completed, Succeeded")
stub.positions = []int{400, 3}
s, _ := newStubSlskd(t, stub)
s.positionPoll = 0
if _, err := s.Grab(
context.Background(),
slskdAlbum(t, stub, "peer", "01 A.flac", "02 B.flac"),
t.TempDir(), nil,
); err != nil {
t.Fatalf("Grab: %v", err)
}
}
// A queue that is moving is progress: the grab outlives the stall timer
// while its position improves.
func TestSlskdAMovingQueueIsProgress(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
// Near enough to wait for, with more improving readings (45) than
// there are queued polls (30), so the queue outlives the stall timer
// (30 polls of at least 2 ms against 40 ms) while still improving,
// however slowly the machine runs the loop.
stub.transfers = queuedThen(30, "Completed, Succeeded")
stub.positions = descending(slskdMaxQueuePosition - 5)
s, _ := newStubSlskd(t, stub)
s.transferPoll = 2 * time.Millisecond
s.positionPoll = 0
s.stallAfter = 40 * time.Millisecond
if _, err := s.Grab(
context.Background(),
slskdAlbum(t, stub, "peer", "01 A.flac", "02 B.flac"),
t.TempDir(), nil,
); err != nil {
t.Fatalf("Grab: %v; a moving queue was treated as a stall", err)
}
}
// However steadily the queue moves, waiting in it has a ceiling.
func TestSlskdQueueHasACeiling(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
stub.transfers = queuedThen(100_000, "Completed, Succeeded")
stub.positions = descending(100_000)
s, _ := newStubSlskd(t, stub)
s.stallAfter = time.Hour
s.queueCeiling = 50 * time.Millisecond
_, err := s.Grab(context.Background(), slskdAlbum(t, stub, "peer"), t.TempDir(), nil)
if !errors.Is(err, ErrSlskdTimeout) {
t.Fatalf("Grab = %v, want the queue ceiling", err)
}
}
+19 -7
View File
@@ -232,12 +232,17 @@ type Candidate struct {
// results give paths and sizes but no tags, so Format and duration are // results give paths and sizes but no tags, so Format and duration are
// inferred from the path and size where possible. // inferred from the path and size where possible.
type CandidateFile struct { type CandidateFile struct {
Path string `json:"path"` Path string `json:"path"`
Size int64 `json:"size"` Size int64 `json:"size"`
Format Format `json:"format"` Format Format `json:"format"`
Bitrate int `json:"bitrate,omitempty"` // kbps, 0 when unknown Bitrate int `json:"bitrate,omitempty"` // kbps, 0 when unknown
IsAudio bool `json:"isAudio"` IsAudio bool `json:"isAudio"`
MatchedTo int `json:"matchedTo,omitempty"` // expected track position
// LengthMillis is the file's duration as the source reports it, or
// 0 when it does not. Soulseek reports it for most audio files.
LengthMillis int64 `json:"lengthMillis,omitempty"`
MatchedTo int `json:"matchedTo,omitempty"` // expected track position
} }
// Format is a normalized audio container/codec name. // Format is a normalized audio container/codec name.
@@ -286,7 +291,14 @@ type MatchScore struct {
TitleFit float64 `json:"titleFit"` // filenames vs expected titles TitleFit float64 `json:"titleFit"` // filenames vs expected titles
ArtistFit float64 `json:"artistFit"` // path/origin vs expected artist ArtistFit float64 `json:"artistFit"` // path/origin vs expected artist
AlbumFit float64 `json:"albumFit"` // folder name vs album title AlbumFit float64 `json:"albumFit"` // folder name vs album title
Completeness float64 `json:"completeness"` // audio files vs expected count Completeness float64 `json:"completeness"` // aligned tracks vs expected count
// DurationFit is how well the aligned files' lengths agree with the
// expected tracks', and DurationKnown whether enough of them stated
// a length for that to count. When it does not, the score is the
// four text signals alone, exactly as before durations were read.
DurationFit float64 `json:"durationFit"`
DurationKnown bool `json:"durationKnown"`
// Anchored records whether an MBID drove this score. Unanchored // Anchored records whether an MBID drove this score. Unanchored
// matches are capped, because there is nothing to be right about. // matches are capped, because there is nothing to be right about.
+88 -11
View File
@@ -1,8 +1,10 @@
package explore package explore
import ( import (
"bytes"
"context" "context"
"database/sql" "database/sql"
"database/sql/driver"
"errors" "errors"
"fmt" "fmt"
"os" "os"
@@ -283,6 +285,25 @@ func (si *SearchIndex) importCoreArtifact(ctx context.Context, path string) erro
} }
merged, mergeErr := si.mergeArtifactRows(ctx, info.rows) merged, mergeErr := si.mergeArtifactRows(ctx, info.rows)
// Every row the artifact declares has to land. The walk partitions
// the artifact's key space, so a total short of info.rows does not
// mean the artifact was smaller than it said -- it means a predicate
// filtered rows out, and the catalog is quietly partial. Equality
// rather than a lower bound because RowsAffected counts an upsert
// that changes nothing, and a row already merged locally is counted
// again here.
//
// One reachable case, so this is not merely a tripwire: a row whose
// mbid is empty is excluded by `mbid > ?` in both encodings, and an
// artifact carrying one would otherwise import as complete.
if mergeErr == nil && merged != info.rows {
mergeErr = fmt.Errorf(
"%w: merged %d of %d rows — a row the artifact holds was not selected",
ErrArtifactUnusable, merged, info.rows,
)
}
if mergeErr == nil { if mergeErr == nil {
si.mergeArtifactCredits(ctx) si.mergeArtifactCredits(ctx)
} }
@@ -348,8 +369,12 @@ func (si *SearchIndex) analyzeIndex() {
// is an index range scan and a cancelled import leaves committed work // is an index range scan and a cancelled import leaves committed work
// behind rather than rolling it all back. // behind rather than rolling it all back.
func (si *SearchIndex) mergeArtifactRows(ctx context.Context, total int) (int, error) { func (si *SearchIndex) mergeArtifactRows(ctx context.Context, total int) (int, error) {
// Asked once, because it is a property of the file and it decides
// how the walk's own comparisons are typed. See artifactKey.
storesText := si.artifactStoresText()
selectColumns := artifactSelectColumns( selectColumns := artifactSelectColumns(
si.artifactStoresText(), si.artifactHasTotals(), storesText, si.artifactHasTotals(),
) )
insertSQL := ` insertSQL := `
@@ -367,7 +392,7 @@ func (si *SearchIndex) mergeArtifactRows(ctx context.Context, total int) (int, e
WHERE mbid > ? AND mbid <= ?` + upsertIndexConflictSQL WHERE mbid > ? AND mbid <= ?` + upsertIndexConflictSQL
var ( var (
cursor string cursor artifactKey
merged int merged int
) )
@@ -376,17 +401,30 @@ func (si *SearchIndex) mergeArtifactRows(ctx context.Context, total int) (int, e
return merged, err return merged, err
} }
upper, hasUpper, err := si.artifactBatchBound(cursor) upper, hasUpper, err := si.artifactBatchBound(storesText, cursor)
if err != nil { if err != nil {
return merged, err return merged, err
} }
if hasUpper && bytes.Compare(upper, cursor) <= 0 {
// The predicate matched the cursor itself, so the walk can
// never advance. SQLite says nothing when a comparison is
// made between types it will not coerce - the query simply
// answers wrongly - so a mismatch here would otherwise spin
// forever behind an unmoving progress bar. Fail instead.
return merged, fmt.Errorf(
"%w: artifact walk did not advance past %x",
ErrArtifactUnusable, []byte(cursor),
)
}
var res sql.Result var res sql.Result
if hasUpper { if hasUpper {
res, err = si.db.ExecContext(insertRangeSQL, cursor, upper) res, err = si.db.ExecContext(insertRangeSQL,
cursor.bind(storesText), upper.bind(storesText))
} else { } else {
res, err = si.db.ExecContext(insertSQL, cursor) res, err = si.db.ExecContext(insertSQL, cursor.bind(storesText))
} }
if err != nil { if err != nil {
@@ -413,26 +451,65 @@ func (si *SearchIndex) mergeArtifactRows(ctx context.Context, total int) (int, e
} }
} }
// artifactKey is one MBID as the attached artifact stores it: 16 raw
// bytes in a compact artifact, the dashed 36-character form in one
// published before that storage change.
//
// It is a type with a bind method rather than a string because the
// comparison it feeds is typed, and the wrong type is silent. SQLite
// does not coerce between TEXT and BLOB and orders every blob after
// every text value, so a cursor bound as text against a byte column
// makes `mbid > ?` true of the whole table - the walk rediscovers the
// same batch bound forever, and `mbid <= ?` false of the whole table,
// so no batch merges at all. Nothing errors; the import simply never
// finishes. bind is the one place that knows which form the column is
// in, decided by artifactStoresText, which asks the artifact rather than
// trusting a version number.
type artifactKey []byte
// bind renders the key as a statement argument in the artifact's own
// encoding.
func (k artifactKey) bind(storesText bool) driver.Value {
if storesText {
return string(k)
}
// Never nil. database/sql converts a nil []byte to SQL NULL, and
// `mbid > NULL` is NULL for every row - so an unset cursor would
// agree with nothing and import nothing, which is the same silently
// empty merge this type exists to prevent, one type over.
if k == nil {
return []byte{}
}
return []byte(k)
}
// artifactBatchBound returns the MBID that ends the next batch, and // artifactBatchBound returns the MBID that ends the next batch, and
// whether one exists — no bound means the remainder is the last batch. // whether one exists — no bound means the remainder is the last batch.
func (si *SearchIndex) artifactBatchBound(cursor string) (string, bool, error) { //
var bound string // The bound is read out of the artifact and handed back as an
// artifactKey, because it becomes the next comparison the walk makes.
func (si *SearchIndex) artifactBatchBound(
storesText bool, cursor artifactKey,
) (artifactKey, bool, error) {
var bound []byte
err := si.db.QueryRowWriter( err := si.db.QueryRowWriter(
`SELECT mbid FROM core.explore_index `SELECT mbid FROM core.explore_index
WHERE mbid > ? ORDER BY mbid LIMIT 1 OFFSET ?`, WHERE mbid > ? ORDER BY mbid LIMIT 1 OFFSET ?`,
cursor, artifactMergeBatch-1, cursor.bind(storesText), artifactMergeBatch-1,
).Scan(&bound) ).Scan(&bound)
if errors.Is(err, sql.ErrNoRows) { if errors.Is(err, sql.ErrNoRows) {
return "", false, nil return nil, false, nil
} }
if err != nil { if err != nil {
return "", false, fmt.Errorf("%w: batch bound: %w", ErrArtifactUnusable, err) return nil, false, fmt.Errorf("%w: batch bound: %w", ErrArtifactUnusable, err)
} }
return bound, true, nil return artifactKey(bound), true, nil
} }
// stampArtifactMeta records what the merge established: the catalog half // stampArtifactMeta records what the merge established: the catalog half
+250 -62
View File
@@ -1,6 +1,7 @@
package explore package explore
import ( import (
"bytes"
"context" "context"
"database/sql" "database/sql"
"encoding/hex" "encoding/hex"
@@ -71,13 +72,7 @@ func writeTestArtifact(
} }
} }
for k, v := range meta { stampArtifactMeta(t, db, meta)
if _, err := db.Exec(
`INSERT INTO artifact_meta (key, value) VALUES (?, ?)`, k, v,
); err != nil {
t.Fatalf("stamp artifact meta: %v", err)
}
}
for _, r := range rows { for _, r := range rows {
if _, err := db.Exec(` if _, err := db.Exec(`
@@ -93,6 +88,101 @@ func writeTestArtifact(
return path return path
} }
// compactArtifactSchema is the artifact cmd/indexexport publishes: the
// catalog's ids as 16 raw bytes, its entity types as codes, and the
// per-release-group total_tracks the exporter added after the first
// artifact was shipped.
//
// It matters that a fixture carries this encoding and not the older
// text one, because SQLite does not coerce between TEXT and BLOB and
// every comparison the importer makes against an mbid is therefore
// encoding-sensitive. writeTestArtifact above is the *other* fixture:
// it still writes the text form, which is what the first published
// artifact carries and what the importer must keep reading.
var compactArtifactSchema = []string{
`CREATE TABLE explore_index (
entity_type INTEGER NOT NULL,
mbid BLOB NOT NULL,
title TEXT NOT NULL,
artist_name TEXT NOT NULL,
artist_mbid BLOB NOT NULL,
aliases TEXT NOT NULL DEFAULT '',
popularity INTEGER NOT NULL DEFAULT 0,
listener_count INTEGER NOT NULL DEFAULT 0,
duration INTEGER NOT NULL DEFAULT 0,
caa_release_mbid BLOB NOT NULL DEFAULT x'',
release_name TEXT NOT NULL DEFAULT '',
primary_type TEXT NOT NULL DEFAULT '',
secondary_types TEXT NOT NULL DEFAULT '',
release_date TEXT NOT NULL DEFAULT '',
total_tracks INTEGER NOT NULL DEFAULT 0,
artist_type TEXT NOT NULL DEFAULT '',
country TEXT NOT NULL DEFAULT '',
disambiguation TEXT NOT NULL DEFAULT '',
sort_name TEXT NOT NULL DEFAULT '',
discog_fetched INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (mbid)
) WITHOUT ROWID`,
`CREATE TABLE artifact_meta (
key TEXT PRIMARY KEY,
value TEXT NOT NULL
)`,
}
// writeCompactTestArtifact builds the artifact the exporter publishes
// today, in its own encoding, so the importer is exercised against what
// a client actually downloads rather than against what it was written
// for.
func writeCompactTestArtifact(
t *testing.T, meta map[string]string, rows []artifactRow,
) string {
t.Helper()
path := filepath.Join(t.TempDir(), "core-index.db")
db, err := sql.Open("sqlite", "file:"+path)
if err != nil {
t.Fatalf("open artifact: %v", err)
}
defer func() { _ = db.Close() }()
for _, stmt := range compactArtifactSchema {
if _, err := db.Exec(stmt); err != nil {
t.Fatalf("create artifact schema: %v", err)
}
}
stampArtifactMeta(t, db, meta)
for _, r := range rows {
if _, err := db.Exec(`
INSERT INTO explore_index
(entity_type, mbid, title, artist_name, artist_mbid, popularity)
VALUES (?, ?, ?, ?, ?, ?)`,
entityCode(r.entityType), mbidBytes(r.mbid), r.title,
r.artistName, mbidBytes(r.artistMBID), r.popularity,
); err != nil {
t.Fatalf("insert artifact row: %v", err)
}
}
return path
}
// stampArtifactMeta writes the artifact_meta rows a fixture declares.
func stampArtifactMeta(t *testing.T, db *sql.DB, meta map[string]string) {
t.Helper()
for k, v := range meta {
if _, err := db.Exec(
`INSERT INTO artifact_meta (key, value) VALUES (?, ?)`, k, v,
); err != nil {
t.Fatalf("stamp artifact meta: %v", err)
}
}
}
// validMeta is the artifact_meta a well-formed artifact carries. // validMeta is the artifact_meta a well-formed artifact carries.
func validMeta() map[string]string { func validMeta() map[string]string {
return map[string]string{ return map[string]string{
@@ -270,6 +360,71 @@ func TestImportCoreArtifactBatchWalkCoversAllRows(t *testing.T) {
} }
} }
// TestImportCoreArtifactBatchWalkCoversAllRowsCompact is the batch walk
// on the encoding the exporter actually publishes.
//
// The walk positions itself by comparing the artifact's own mbid column
// against the last id it reached, and that column holds 16 raw bytes.
// SQLite does not coerce between TEXT and BLOB, and a blob sorts after
// every text value, so a cursor bound as text is a predicate that either
// matches every row or none: `mbid > ?` with an empty text key is true
// of the whole table, so
// the 100th row is always the 100th row and the bound never advances,
// while `mbid <= <text>` is false of the whole table, so no batch ever
// merges. The result is not a wrong import but an unbounded loop that
// merges nothing and never fails.
//
// Both encodings are covered on purpose. The walk was only ever tested
// against the text fixture above, which is why it shipped broken on the
// one the clients download.
func TestImportCoreArtifactBatchWalkCoversAllRowsCompact(t *testing.T) {
db := database.NewTestDB(t)
si := NewSearchIndex(db, nil, nil, testLogger())
original := artifactMergeBatch
artifactMergeBatch = 100
t.Cleanup(func() { artifactMergeBatch = original })
const total = 337
rows := make([]artifactRow, 0, total)
for i := range total {
rows = append(rows, artifactRow{
entityType: EntityRecording,
mbid: syntheticMBID(i),
title: "Song",
artistName: "Artist",
artistMBID: artA,
popularity: i,
})
}
path := writeCompactTestArtifact(t, validMeta(), rows)
if err := si.importCoreArtifact(context.Background(), path); err != nil {
t.Fatalf("importCoreArtifact: %v", err)
}
var got, top int
if err := db.QueryRowWriter(
"SELECT COUNT(*), MAX(popularity) FROM explore_index",
).Scan(&got, &top); err != nil {
t.Fatalf("count rows: %v", err)
}
if got != total {
t.Errorf("merged %d rows, want %d", got, total)
}
// A count alone would pass if the walk re-merged the same first
// batch forever, so the far end of the artifact is checked too.
if top != total-1 {
t.Errorf("highest popularity = %d, want %d", top, total-1)
}
}
func TestImportCoreArtifactRejectsBadArtifacts(t *testing.T) { func TestImportCoreArtifactRejectsBadArtifacts(t *testing.T) {
tests := []struct { tests := []struct {
name string name string
@@ -454,61 +609,9 @@ func TestArtifactColumnsMatchExporter(t *testing.T) {
// the importer decides by asking the artifact, not by trusting a // the importer decides by asking the artifact, not by trusting a
// version number, and both must land identically. // version number, and both must land identically.
func TestImportCoreArtifactAcceptsBothEncodings(t *testing.T) { func TestImportCoreArtifactAcceptsBothEncodings(t *testing.T) {
compact := filepath.Join(t.TempDir(), "core-index.db") compact := writeCompactTestArtifact(t, validMeta(), []artifactRow{
{EntityArtist, artA, "Artist A", "Artist A", artA, 5000},
db, err := sql.Open("sqlite", "file:"+compact) })
if err != nil {
t.Fatalf("open artifact: %v", err)
}
if _, err := db.Exec(`CREATE TABLE explore_index (
entity_type INTEGER NOT NULL,
mbid BLOB NOT NULL,
title TEXT NOT NULL,
artist_name TEXT NOT NULL,
artist_mbid BLOB NOT NULL,
aliases TEXT NOT NULL DEFAULT '',
popularity INTEGER NOT NULL DEFAULT 0,
listener_count INTEGER NOT NULL DEFAULT 0,
duration INTEGER NOT NULL DEFAULT 0,
caa_release_mbid BLOB NOT NULL DEFAULT x'',
release_name TEXT NOT NULL DEFAULT '',
primary_type TEXT NOT NULL DEFAULT '',
secondary_types TEXT NOT NULL DEFAULT '',
release_date TEXT NOT NULL DEFAULT '',
artist_type TEXT NOT NULL DEFAULT '',
country TEXT NOT NULL DEFAULT '',
disambiguation TEXT NOT NULL DEFAULT '',
sort_name TEXT NOT NULL DEFAULT '',
discog_fetched INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (mbid)
)`); err != nil {
t.Fatalf("create artifact table: %v", err)
}
if _, err := db.Exec(
`CREATE TABLE artifact_meta (key TEXT PRIMARY KEY, value TEXT NOT NULL)`,
); err != nil {
t.Fatalf("create artifact meta: %v", err)
}
for k, v := range validMeta() {
if _, err := db.Exec(
"INSERT INTO artifact_meta (key, value) VALUES (?, ?)", k, v,
); err != nil {
t.Fatalf("write artifact meta: %v", err)
}
}
if _, err := db.Exec(`
INSERT INTO explore_index (entity_type, mbid, title, artist_name, artist_mbid, popularity)
VALUES (1, ?, 'Artist A', 'Artist A', ?, 5000)`,
mbidBytes(artA), mbidBytes(artA),
); err != nil {
t.Fatalf("write artifact row: %v", err)
}
_ = db.Close()
live := database.NewTestDB(t) live := database.NewTestDB(t)
si := NewSearchIndex(live, nil, nil, testLogger()) si := NewSearchIndex(live, nil, nil, testLogger())
@@ -748,3 +851,88 @@ func TestImportCoreArtifactWithoutCredits(t *testing.T) {
t.Errorf("credit refs = %d, want 0", refs) t.Errorf("credit refs = %d, want 0", refs)
} }
} }
// TestImportCoreArtifactRefusesAMergeThatLosesRows is the count guard's
// positive case.
//
// The walk's predicates partition the artifact's key space, so a merge
// that lands fewer rows than the artifact declares means a predicate
// dropped some — and the failure is a catalog that looks populated and
// is missing things nobody can name. An empty mbid is the reachable
// way to get there: `mbid > ?` is false of it in both encodings, so it
// is never selected, and nothing else in the import would notice.
func TestImportCoreArtifactRefusesAMergeThatLosesRows(t *testing.T) {
db := database.NewTestDB(t)
si := NewSearchIndex(db, nil, nil, testLogger())
path := writeCompactTestArtifact(t, validMeta(), []artifactRow{
{EntityArtist, artA, "Artist A", "Artist A", artA, 5000},
{EntityArtist, "", "Nameless", "Artist A", artA, 4000},
})
err := si.importCoreArtifact(context.Background(), path)
if err == nil {
t.Fatal("a merge that lost a row was reported as a complete import")
}
if !strings.Contains(err.Error(), "merged 1 of 2 rows") {
t.Errorf("error = %v, want it to name the shortfall", err)
}
// And the same rule as every other rejection: a failed merge must not
// leave the index claiming it has a catalog, or the real build would
// never run again.
if si.hasMeta(dumpImportDoneKey) {
t.Error("a failed import still stamped dump_import_done")
}
}
// TestArtifactKeyBindsInTheArtifactsOwnEncoding pins the one place the
// batch walk's comparison type is decided.
//
// Every wrong answer is silent, which is why it is worth pinning all
// four. SQLite does not coerce TEXT to BLOB and orders every blob after
// every text value, so a text key against a byte column makes
// `mbid > ?` true of the whole artifact - the cursor never advances and
// the walk spins forever without merging a row - while a byte key
// against a text column makes it false of the whole artifact, so every
// batch merges nothing and the import "succeeds" empty. An unset cursor
// is the same fault once more: database/sql converts a nil []byte to
// SQL NULL, and `mbid > NULL` matches no row at all.
func TestArtifactKeyBindsInTheArtifactsOwnEncoding(t *testing.T) {
raw := mbidBytes(artA)
for _, tt := range []struct {
name string
key artifactKey
want []byte
}{
{"unset", nil, []byte{}},
{"set", artifactKey(raw), raw},
} {
t.Run("bytes/"+tt.name, func(t *testing.T) {
got, ok := tt.key.bind(false).([]byte)
if !ok {
t.Fatalf("bind(false) = %T, want []byte", tt.key.bind(false))
}
if got == nil {
t.Fatal("bound to SQL NULL, which matches no row")
}
if !bytes.Equal(got, tt.want) {
t.Errorf("bind(false) = %x, want %x", got, tt.want)
}
})
}
// The dashed form is what an artifact published before the storage
// change carries, and it has to compare as text against text.
if got := artifactKey(nil).bind(true); got != "" {
t.Errorf("bind(true) on an unset cursor = %#v, want an empty string", got)
}
if got := artifactKey(artA).bind(true); got != artA {
t.Errorf("bind(true) = %#v, want %q", got, artA)
}
}
+266
View File
@@ -0,0 +1,266 @@
package explore
import (
"context"
"database/sql"
"io"
"os"
"path/filepath"
"strings"
"testing"
"yellowjacket/backend/database"
)
// Import of the artifact we actually publish, as a client imports it.
//
// Every other test here builds a fixture, and a fixture is a second
// description of the storage format that can be wrong in the same
// direction as the code reading it. That is how #258 shipped: the
// importer positioned its batch walk with a Go `string` cursor against
// the artifact's 16-byte `mbid` column, and SQLite neither coerces
// between TEXT and BLOB nor complains about the comparison — so the walk
// merged nothing and never advanced, and no install could finish its
// first index build. The fixture that guards the walk writes the old
// text encoding; the only compact fixture is one row, below the batch
// size, so the bound query never ran. Both passed throughout.
//
// So this one takes the published file and runs the client's own path
// over it — checksum, decompress, merge — and asserts that what the
// artifact holds is what the client ends up with.
//
// It skips without the path, so an ordinary test run pays nothing for
// it, and the publish job is where it is meant to run:
//
// YJ_CORE_INDEX_ARTIFACT=/tmp/core-index.db.zst \
// go test -tags indexbuild -run TestImportPublishedArtifact \
// ./backend/explore/
//
// The indexbuild tag is not incidental: that job's container has no GTK,
// and the default tag set links the app through Wails.
// publishedArtifactEnv points at the published artifact: the compressed
// core-index.db.zst, or the unpacked core-index.db.
const publishedArtifactEnv = "YJ_CORE_INDEX_ARTIFACT"
// artifactTotals is the pair this test compares across the boundary.
//
// Rows is the whole point — a merge that lands fewer of them than the
// artifact declares is a catalog that looks populated and is missing
// things nobody can name — and popularity is the half whose absence was
// reported when it happened, because it arrives only through the merge.
type artifactTotals struct {
rows int
withListen int
}
func TestImportPublishedArtifact(t *testing.T) {
published := strings.TrimSpace(os.Getenv(publishedArtifactEnv))
if published == "" {
t.Skipf("set %s=<core-index.db.zst> to import the published artifact",
publishedArtifactEnv)
}
if _, err := os.Stat(published); err != nil {
t.Fatalf("%s: %v", publishedArtifactEnv, err)
}
// A file-backed database rather than NewTestDB's in-memory one: the
// artifact is ~135MB and a million rows, which is not a thing to hold
// in RAM inside a test. YJ_HOME is how NewDB is pointed somewhere
// disposable, and going through NewDB means this is the constructor,
// the schema and the read pool the app itself opens.
//
// Nothing closes it, because nothing can: `DB` has no Close and the
// app's handles are process-lifetime by design. The directory is
// unlinked at cleanup and the file goes with it.
t.Setenv("YJ_HOME", t.TempDir())
db, err := database.NewDB(testLogger())
if err != nil {
t.Fatalf("open database: %v", err)
}
si := NewSearchIndex(db, nil, nil, testLogger())
// The checksum the publisher shipped, if it shipped one. Every
// client verifies it and refuses the artifact when it does not
// match, so a wrong one breaks Explore for everyone who has not
// already imported — and nothing else would see it, because the
// comparison is between two files only the publisher has.
if want, ok := publishedChecksum(published); ok {
got, err := fileSHA256(published)
if err != nil {
t.Fatalf("checksum the artifact: %v", err)
}
if got != want {
t.Errorf("published artifact hashes to %s, but its .sha256 says %s",
got, want)
}
}
unpacked := unpackPublishedArtifact(t, si, published)
want, err := artifactTotalsOf(unpacked)
if err != nil {
t.Fatalf("count the artifact's rows: %v", err)
}
if err := si.importCoreArtifact(context.Background(), unpacked); err != nil {
t.Fatalf("importCoreArtifact: %v", err)
}
got, err := indexTotalsOf(db)
if err != nil {
t.Fatalf("count the index's rows: %v", err)
}
if got.rows != want.rows {
t.Errorf("merged %d rows, but the artifact holds %d",
got.rows, want.rows)
}
if got.withListen != want.withListen {
t.Errorf("%d rows carry a listen count, but the artifact holds %d of them",
got.withListen, want.withListen)
}
// The FTS index is rebuilt from the table once the merge is done, and
// it is what search actually reads: a merge that lands without it
// leaves Explore silently matching nothing, which is the state #258
// produced by a different route.
var indexed int
if err := db.QueryRowWriter(
"SELECT COUNT(*) FROM explore_index_fts",
).Scan(&indexed); err != nil {
t.Fatalf("count the FTS index: %v", err)
}
if indexed != got.rows {
t.Errorf("FTS index holds %d rows against the table's %d",
indexed, got.rows)
}
// And one row read back through the app's own path, which is the
// other direction of every conversion the merge makes: a byte MBID
// out of the table, the app's dashed form, and back in as a lookup.
var raw []byte
if err := db.QueryRowWriter(`
SELECT mbid FROM explore_index
WHERE entity_type = 1 /* artist */ AND popularity > 0
ORDER BY popularity DESC LIMIT 1`).Scan(&raw); err != nil {
t.Fatalf("read a stored mbid: %v", err)
}
dashed, err := mbidFromBytes(raw)
if err != nil {
t.Fatalf("the stored mbid is not one: %v", err)
}
artist := si.LookupArtistByMBID(dashed)
if artist == nil {
t.Fatalf("the artifact's most popular artist %s does not look up", dashed)
}
if artist.Popularity == 0 {
t.Errorf("artist %s came back with no popularity", dashed)
}
}
// publishedChecksum reads the sha256 the publisher wrote beside the
// artifact, in `sha256sum` output form. A missing file is not a
// failure: it is only there when the artifact came from the publish job.
func publishedChecksum(path string) (string, bool) {
body, err := os.ReadFile(path + ".sha256")
if err != nil {
return "", false
}
sum := strings.TrimSpace(string(body))
if i := strings.IndexAny(sum, " \t"); i > 0 {
sum = sum[:i]
}
if len(sum) != 64 {
return "", false
}
return strings.ToLower(sum), true
}
// unpackPublishedArtifact returns a path to the unpacked database,
// going through the client's own decompression when it is handed the
// compressed file that is actually published.
func unpackPublishedArtifact(t *testing.T, si *SearchIndex, path string) string {
t.Helper()
if strings.HasSuffix(path, ".db") {
return path
}
// Copied into the test's own directory first: decompress writes
// beside the compressed file, and the publisher's directory is not
// this test's to write in.
staging := t.TempDir()
dst := filepath.Join(staging, coreArtifactFile)
src, err := os.Open(path)
if err != nil {
t.Fatalf("open the published artifact: %v", err)
}
defer func() { _ = src.Close() }()
out, err := os.Create(dst)
if err != nil {
t.Fatalf("create a staging copy: %v", err)
}
if _, err := io.Copy(out, src); err != nil {
t.Fatalf("copy the published artifact: %v", err)
}
if err := out.Close(); err != nil {
t.Fatalf("close the staging copy: %v", err)
}
fetcher := &artifactFetcher{si: si, stagingDir: staging}
if err := fetcher.decompress(context.Background()); err != nil {
t.Fatalf("decompress the published artifact: %v", err)
}
return fetcher.unpackedPath()
}
// artifactTotalsOf counts what an artifact file holds, read directly so
// the numbers do not depend on anything the client does.
func artifactTotalsOf(path string) (artifactTotals, error) {
db, err := sql.Open("sqlite", "file:"+path+"?mode=ro")
if err != nil {
return artifactTotals{}, err
}
defer func() { _ = db.Close() }()
var totals artifactTotals
err = db.QueryRow(`SELECT COUNT(*), COALESCE(SUM(popularity > 0), 0)
FROM explore_index`).Scan(&totals.rows, &totals.withListen)
if err != nil {
return artifactTotals{}, err
}
return totals, nil
}
// indexTotalsOf counts what the client ended up with.
func indexTotalsOf(db *database.DB) (artifactTotals, error) {
var totals artifactTotals
err := db.QueryRowWriter(`SELECT COUNT(*), COALESCE(SUM(popularity > 0), 0)
FROM explore_index`).Scan(&totals.rows, &totals.withListen)
return totals, err
}
+4 -4
View File
@@ -8,6 +8,7 @@ import (
"yellowjacket/backend/coverart" "yellowjacket/backend/coverart"
"yellowjacket/backend/database/sql/sqlcgen" "yellowjacket/backend/database/sql/sqlcgen"
"yellowjacket/internal/testfixtures"
) )
// TestScan_StoresOnlyCoverTiers pins the size decision: a scan writes // TestScan_StoresOnlyCoverTiers pins the size decision: a scan writes
@@ -26,10 +27,9 @@ func TestScan_StoresOnlyCoverTiers(t *testing.T) {
lib, db := setupTestLibrary(t) lib, db := setupTestLibrary(t)
root, err := filepath.Abs("../../test_data/music_library_test") // Load skips when the fixture library has not been generated, as
if err != nil { // every other fixture test does.
t.Fatalf("resolve fixture path: %v", err) root := testfixtures.Load(t).Root()
}
library, err := db.Queries.CreateLibrary(lib.ctx, sqlcgen.CreateLibraryParams{ library, err := db.Queries.CreateLibrary(lib.ctx, sqlcgen.CreateLibraryParams{
Name: "Fixtures", Name: "Fixtures",
+17
View File
@@ -60,6 +60,23 @@ const PHONE = { width: 424, height: 439 };
const DESKTOP = { width: 1100, height: 800 }; const DESKTOP = { width: 1100, height: 800 };
test.describe('background jobs on a phone', () => { test.describe('background jobs on a phone', () => {
/**
* **State a spec stages is the spec's to clear.** `/__test/emit` writes
* to a store nothing resets, so the event that staged a job is the
* event that clears it — `JobStore` replaces its whole list from every
* snapshot, so `testctl` needs no special case.
*
* **Measured on #168: this does not currently outlive the page.** Every
* test gets a fresh page, and `JobStore.init()` refetches `GetJobs()`
* from a backend registry that `/__test/emit` never writes to, so the
* staged job is gone before the next spec starts. Ownership is stated
* rather than a live leak repaired — the leak needs a page that
* survives its own spec, and there is none today.
*/
test.afterEach(async ({ testctl }) => {
await testctl.emit('JobsChanged', []);
});
test('are shown in the band, without opening anything', async ({ test('are shown in the band, without opening anything', async ({
app, app,
testctl, testctl,
+2 -2
View File
@@ -171,8 +171,8 @@ test.describe('search on a phone', () => {
// Attached, not visible: `wa-dialog`'s host is `display: contents`, // Attached, not visible: `wa-dialog`'s host is `display: contents`,
// so the element carrying the testid always reports hidden — what // so the element carrying the testid always reports hidden — what
// is visible is the native `<dialog>` inside it. That awkwardness // is visible is the native `<dialog>` inside it. That awkwardness
// is written down in CLAUDE.md and is why the assertion that this // is why the assertion that this is really up is the role query
// is really up is the role query below. // below.
await expect(dialog).toBeAttached(); await expect(dialog).toBeAttached();
// Named, which `getByRole` can answer and the a11y snapshot cannot // Named, which `getByRole` can answer and the a11y snapshot cannot
+16
View File
@@ -102,6 +102,22 @@ const collapsed = (page: Page) =>
})); }));
test.describe('the top bar fits the window', () => { test.describe('the top bar fits the window', () => {
/**
* **State a spec stages is the spec's to clear** (#168). `/__test/emit`
* writes to a store nothing resets, and this file stages the widest job
* in the app, so it puts it back — with the same event, since the store
* replaces its whole list from every snapshot.
*
* **Measured: it does not currently outlive the page.** Every test gets
* a fresh page and `JobStore.init()` refetches `GetJobs()` from a
* backend registry `/__test/emit` never writes to, so nothing is being
* repaired here; the rule is stated because it costs one line and the
* leak would need only one spec that keeps a page alive.
*/
test.afterEach(async ({ testctl }) => {
await testctl.emit('JobsChanged', []);
});
/** /**
* The phone's answer, which is not "it fits" (#57). * The phone's answer, which is not "it fits" (#57).
* *
@@ -121,6 +121,12 @@ export interface CandidateFile {
"bitrate"?: number; "bitrate"?: number;
"isAudio": boolean; "isAudio": boolean;
/**
* LengthMillis is the file's duration as the source reports it, or
* 0 when it does not. Soulseek reports it for most audio files.
*/
"lengthMillis"?: number;
/** /**
* expected track position * expected track position
*/ */
@@ -453,10 +459,19 @@ export interface MatchScore {
"albumFit": number; "albumFit": number;
/** /**
* audio files vs expected count * aligned tracks vs expected count
*/ */
"completeness": number; "completeness": number;
/**
* DurationFit is how well the aligned files' lengths agree with the
* expected tracks', and DurationKnown whether enough of them stated
* a length for that to count. When it does not, the score is the
* four text signals alone, exactly as before durations were read.
*/
"durationFit": number;
"durationKnown": boolean;
/** /**
* Anchored records whether an MBID drove this score. Unanchored * Anchored records whether an MBID drove this score. Unanchored
* matches are capped, because there is nothing to be right about. * matches are capped, because there is nothing to be right about.
@@ -172,8 +172,7 @@ describe('<player-progress-line>', () => {
* The reason this component asks `matchMedia` instead of letting a * The reason this component asks `matchMedia` instead of letting a
* stylesheet hide it: a media query cannot stop a 1 Hz interval * stylesheet hide it: a media query cannot stop a 1 Hz interval
* running for the life of every desktop session. That claim is * running for the life of every desktop session. That claim is
* load-bearing in CLAUDE.md, so it is asserted rather than * load-bearing, so it is asserted rather than described — the timer count, because a desktop render is empty
* described — the timer count, because a desktop render is empty
* either way and so cannot tell the two apart. * either way and so cannot tell the two apart.
*/ */
it('runs no interpolation timer above the breakpoint', async () => { it('runs no interpolation timer above the breakpoint', async () => {
+2 -2
View File
@@ -1,6 +1,6 @@
module yellowjacket module yellowjacket
go 1.25.0 go 1.26
require ( require (
github.com/BurntSushi/toml v1.6.0 github.com/BurntSushi/toml v1.6.0
@@ -144,7 +144,7 @@ require (
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect
github.com/go-git/go-billy/v5 v5.9.0 // indirect github.com/go-git/go-billy/v5 v5.9.0 // indirect
github.com/go-git/go-git/v5 v5.19.2 // indirect github.com/go-git/go-git/v5 v5.19.2 // indirect
github.com/go-json-experiment/json v0.0.0-20251027170946-4849db3c2f7e // indirect github.com/go-json-experiment/json v0.0.0-20260820222146-c27c302e5fc3 // indirect
github.com/go-ole/go-ole v1.3.0 // indirect github.com/go-ole/go-ole v1.3.0 // indirect
github.com/go-resty/resty/v2 v2.17.1 // indirect github.com/go-resty/resty/v2 v2.17.1 // indirect
github.com/go-sql-driver/mysql v1.9.3 // indirect github.com/go-sql-driver/mysql v1.9.3 // indirect
+2 -2
View File
@@ -362,8 +362,8 @@ github.com/go-git/go-git/v5 v5.19.2/go.mod h1:QqCBE1EFN5ddFmrliLQ3/ntRCUjZU3EJuw
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU= github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8= github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8= github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-json-experiment/json v0.0.0-20251027170946-4849db3c2f7e h1:Lf/gRkoycfOBPa42vU2bbgPurFong6zXeFtPoxholzU= github.com/go-json-experiment/json v0.0.0-20260820222146-c27c302e5fc3 h1:UADEEmDKgfXbtnGJZ97beY5XLo9ZechG1nlU4KnRrkE=
github.com/go-json-experiment/json v0.0.0-20251027170946-4849db3c2f7e/go.mod h1:uNVvRXArCGbZ508SxYYTC5v1JWoz2voff5pm25jU1Ok= github.com/go-json-experiment/json v0.0.0-20260820222146-c27c302e5fc3/go.mod h1:tphK2c80bpPhMOI4v6bIc2xWywPfbqi1Z06+RcrMkDg=
github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as= github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as= github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
github.com/go-kit/log v0.1.0/go.mod h1:zbhenjAZHb184qTLMA9ZjW7ThYL0H2mk7Q6pNt4vbaY= github.com/go-kit/log v0.1.0/go.mod h1:zbhenjAZHb184qTLMA9ZjW7ThYL0H2mk7Q6pNt4vbaY=
+1 -2
View File
@@ -64,8 +64,7 @@ var frontendDistAssets embed.FS
// Returning early is not a degraded mode: `nativeInit` has already // Returning early is not a degraded mode: `nativeInit` has already
// re-attached the bridge, so the recreated activity's WebView talks to // re-attached the bridge, so the recreated activity's WebView talks to
// the app that is still running, with its queue and its playback // the app that is still running, with its queue and its playback
// position intact. See CLAUDE.md, "An activity is a view onto the // position intact.
// process".
// //
// It is inert off Android, where a process has exactly one main(). // It is inert off Android, where a process has exactly one main().
var mainStarted atomic.Bool var mainStarted atomic.Bool
+1 -1
View File
@@ -18,7 +18,7 @@ arch=('x86_64')
url="https://git.ljones.me/yonlu/yellowjacket" url="https://git.ljones.me/yonlu/yellowjacket"
license=('custom') license=('custom')
depends=('webkitgtk-6.0' 'gtk4' 'alsa-lib' 'hicolor-icon-theme') depends=('webkitgtk-6.0' 'gtk4' 'alsa-lib' 'hicolor-icon-theme')
makedepends=('go>=1.25' 'nodejs>=22' 'pnpm' 'git') makedepends=('go>=1.26' 'nodejs>=22' 'pnpm' 'git')
options=('!lto') options=('!lto')
# Source is overridable so the same PKGBUILD works two ways: # Source is overridable so the same PKGBUILD works two ways:
+62 -7
View File
@@ -82,23 +82,69 @@ targets="$({ make -pqRr 2>/dev/null || true; } |
# happened to break there, and a check that fails on reflow gets # happened to break there, and a check that fails on reflow gets
# disabled rather than fixed. # disabled rather than fixed.
# #
# **An inline span may be hard-wrapped, and then the mention is split
# across two lines.** `make` at the end of one line and its target at
# the start of the next is one code span to Markdown and two strings to
# a per-line regex, so the target was invisible — and these docs are
# mostly hard-wrapped prose, so the wrap is what the author does not
# think about. Lines are therefore joined while the span is still open,
# which is what an odd number of backticks means.
#
# Joining re-opens the reflow trap above unless it is bounded, so it is
# bounded three ways: a fence flushes first (a fenced command is already
# whole, and joining inside one would break the line-start rule), a
# blank line flushes (CommonMark does not allow a blank line inside a
# code span, so nothing legitimate is split by one), and so does a file
# boundary. A stray odd backtick in prose therefore costs one paragraph
# of over-matching rather than the rest of the file.
#
# AGENTS.md is deliberately not in this list: it is a symlink to # AGENTS.md is deliberately not in this list: it is a symlink to
# CLAUDE.md, asserted above, so scanning it would report every failure # CLAUDE.md, asserted above, so scanning it would report every failure
# twice under two names. # twice under two names.
mentioned="$(printf '%s\n' "$docs" | mentioned="$(printf '%s\n' "$docs" |
xargs awk ' xargs awk '
FNR == 1 { fence = 0 } function scan(text, rest) {
/^```/ { fence = !fence; next } rest = text
{
rest = $0
while (match(rest, /`make [a-z][a-z0-9-]*/)) { while (match(rest, /`make [a-z][a-z0-9-]*/)) {
print substr(rest, RSTART + 6, RLENGTH - 6) print substr(rest, RSTART + 6, RLENGTH - 6)
rest = substr(rest, RSTART + RLENGTH) rest = substr(rest, RSTART + RLENGTH)
} }
if (fence && match($0, /^make [a-z][a-z0-9-]*/)) { }
print substr($0, 6, RLENGTH - 5)
function lineStart(text) {
if (match(text, /^make [a-z][a-z0-9-]*/)) {
print substr(text, 6, RLENGTH - 5)
} }
} }
function ticks(s, n, i) {
n = 0
for (i = 1; i <= length(s); i++) {
if (substr(s, i, 1) == "`") n++
}
return n
}
function flush() {
if (buf == "") return
scan(buf)
if (fence) lineStart(buf)
buf = ""
}
FNR == 1 { flush(); fence = 0 }
/^```/ { flush(); fence = !fence; next }
/^[[:space:]]*$/ { flush(); next }
{
if (fence) { scan($0); lineStart($0); next }
buf = (buf == "" ? $0 : buf " " $0)
if (ticks(buf) % 2 == 0) flush()
}
END { flush() }
' | sort -u)" ' | sort -u)"
missing="" missing=""
@@ -113,7 +159,16 @@ if [ -n "$missing" ]; then
echo "skill-check: the docs name make targets that do not exist:" >&2 echo "skill-check: the docs name make targets that do not exist:" >&2
for t in $missing; do for t in $missing; do
echo " make $t" >&2 echo " make $t" >&2
printf '%s\n' "$docs" | xargs grep -ln "make $t" | sed 's/^/ /' >&2 # `make <t>` on one line first, because that is where a target is
# normally named and it is the precise answer. The bare name is the
# fallback, and it exists because the parser above can now find a
# mention that *this* grep cannot: a wrapped span has `make` and its
# target on different lines. Without it a missing target reported no
# file at all, and `set -o pipefail` turned the empty grep into exit
# 123, before the line telling the author what to do.
hits="$(printf '%s\n' "$docs" | xargs grep -ln "make $t" 2>/dev/null || true)"
[ -n "$hits" ] || hits="$(printf '%s\n' "$docs" | xargs grep -ln -- "$t" 2>/dev/null || true)"
[ -n "$hits" ] && printf '%s\n' "$hits" | sed 's/^/ /' >&2
done done
echo "Fix the docs, or restore the target." >&2 echo "Fix the docs, or restore the target." >&2
exit 1 exit 1