Compare commits

..
Author SHA1 Message Date
yonluandClaude Opus 5.5 399dcc05b3 docs(notes): record slskd's API as read from its source
CI / check (push) Skipped
CI / e2e (push) Skipped
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017HJiuc3ZZhxsPXz3ozTirT
2026-09-26 22:11:30 -04:00
yonluandClaude Opus 5.5 ef89707bb1 feat(download): fill in the tracks a nearly complete album is missing
A grab that delivers nine of twelve tracks clears the completeness floor
and is imported, and the other three were never looked for. On Soulseek
that is the commonest way an album ends up almost right: one peer's
folder lacks a track, or one file fails.

After a successful import the manager now compares the tracks the files
were aligned to (ImportResult.Matched, new) with the expected tracklist.
When one to three are missing, and fewer than half, it searches for each
one on its own, as the track's artist and title with the album kept for
ranking. It grabs the first auto-acceptable copy that is not from the
source that already failed to supply it, and is not a delegate, which
would place it in its own library. The candidate is trimmed to the one
file aligned to the track. The import uses the album's own request with
ImportOptions.Only, which skips the completeness check and imports only
a file aligned to the missing track, so it is tagged and placed as part
of the album, and anything else the folder brought is left out.

One attempt per track, and nothing here fails the download: the album
is already imported, so a track that cannot be found is logged on the
job and left. slskd accepts one-file folders for a request that expects
one track, which the per-track search needs.

Closes #276

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017HJiuc3ZZhxsPXz3ozTirT
2026-09-26 22:11:03 -04:00
yonluandClaude Opus 5.5 c88fc1c7c7 feat(download): judge a queued slskd peer by its queue position
No bytes for ten minutes usually means a peer has queued us, and the
stall timer could not tell position 2 from position 400: the first was
abandoned while it was about to start, the second was waited on for ten
minutes for nothing.

While a requested file is "Queued, Remotely", the grab asks slskd for
its place (GET .../downloads/{user}/{id}/position, which asks the peer)
once a minute. A place that improved counts as progress and restarts
the stall clock. A place beyond 50 twice running gives the peer up at
once, and the manager moves to the next copy; two readings because
slskd documents the figure as possibly inaccurate. Waiting in a queue
has an overall ceiling of an hour without a byte, since a queue moving
one place an hour would otherwise hold the grab all day. As with a
stall, files that already arrived still go forward.

Closes #275

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017HJiuc3ZZhxsPXz3ozTirT
2026-09-26 22:07:18 -04:00
yonluandClaude Opus 5.5 7a9dd69d30 fix(download): own folder per slskd grab, search timeout in seconds
Checked against slskd 0.26.0's source rather than a live daemon, which
#267 never had.

slskd reads a search's searchTimeout in seconds, counted from the last
response. We sent milliseconds, telling it a search may idle for five
hours, so a search never completed on its own. It now sends seconds,
with slskd's floor of 5.

slskd writes a finished file to <downloads>/<remote leaf folder>/, and
when a name is taken it writes name_<ticks>.ext beside it. collect found
files by name there, so a file left by an earlier failed attempt, or by
the user's own download, was collected in place of this grab's. A user
who changed slskd's destination setting got nothing collected at all.

slskd 0.26 takes a batch download with an explicit destination. Each
grab now enqueues batches into yellowjacket/<uuid>/ (one per disc, since
a batch's files land flat), collects from exactly there, and removes the
folder afterwards, including after a failure. An older daemon answers
the batch route with 400, which is remembered, and the per-user enqueue
is used. There, collect skips files that were already present,
unchanged, before the enqueue, and takes the renamed copy slskd wrote
instead. The comparison is against a snapshot, not a clock, because
slskd may run on another machine. The per-folder lock from #272 is kept
only while batches are not known to work.

The test stub now writes files when they are enqueued, as slskd does,
including the rename, so tests no longer stage files before a grab.
An opt-in TestSlskdLive runs against a real daemon when YJ_SLSKD_URL,
YJ_SLSKD_API_KEY and YJ_SLSKD_DOWNLOADS are set.

Closes #274

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017HJiuc3ZZhxsPXz3ozTirT
2026-09-26 22:04:50 -04:00
89 changed files with 2406 additions and 3386 deletions

No files matched your search

@@ -647,7 +647,7 @@ window.__yj = { call(name, args) {
That turns the device into a tier that can be *driven* rather than only That turns the device into a tier that can be *driven* rather than only
looked at — `__yj.call("player.Player.LoadFile", [path])` and looked at — `__yj.call("player.Player.LoadFile", [path])` and
`__yj.call("library.Library.AddLibrary", ["/sdcard/Music/..."])` are how `__yj.call("library.Library.AddLibrary", ["/sdcard/Music/..."])` are how
#53 was measured. Names are the Go ones (`GetTrackTable`, not #53 was measured. Names are the Go ones (`GetTracks`, not
`GetAllTracks`); an unknown one comes back as a plain `GetAllTracks`); an unknown one comes back as a plain
`unknown bound method name`, so a wrong guess is loud. `unknown bound method name`, so a wrong guess is loud.
+30 -81
View File
@@ -5098,86 +5098,35 @@ beside what they explain. These three did not:
full-screen surface on a 424px viewport; back and a 44px close button full-screen surface on a 424px viewport; back and a 44px close button
answer it instead. answer it instead.
## What the library payloads actually cost (measured 2026-10-05, desktop dev build) ## slskd's API, read from its source rather than a live daemon (2026-09-26)
Plan 023 (#279–#284). Measured against `make sandbox-seed-bulk` `backend/download/provider_slskd.go` had never run against a real slskd
(50 000 tracks, 371 MB `yj.db` in the smaller copy) with the desktop when #263–#272 shipped, so its assumptions were checked against slskd
dev build and `e2e/perf/measure.mjs`, which grew a `memory` section for 0.26.0's source. One was wrong, and one design was only safe by luck.
it: backend RSS and peak from `/proc`, the Go heap from pprof, the These are properties of someone else's server; re-check on an upgrade.
page's JS heap and DOM counters from CDP, and the bytes every binding `TestSlskdLive` (env-gated, see its comment) is the way to confirm them
returned, before and after the first open of Tracks. against a running one.
| | before | after #281 | after #280 | - **`searchTimeout` is seconds, from the last response**, minimum 5
|---|---|---|---| (`SearchRequest.cs`). We sent milliseconds (#274). The other search
| Backend RSS at rest | 543 MB | 296 MB | 189 MB | options — `responseLimit`, `fileLimit`, `filterResponses`,
| Backend peak RSS | 571 MB | 337 MB | 281 MB | `minimumResponseFileCount`, `maximumPeerQueueLength` — are named as we
| Go heap held | 361 MB | 125 MB | 7 MB | send them; slskd's defaults are 100 responses, 10 000 files, queue
| JS heap at rest | 31.8 MB | 18.0 MB | 5.2 MB | 1 000 000.
| Binding bytes at rest | 35.9 MB | 12.1 MB | 1.7 MB | - **`GET /searches/{id}/responses` exists**, and `DELETE
| JS heap after a browse | 36.5 MB | 22.7 MB | 23.1 MB | /transfers/downloads/{user}/{id}?remove=true` cancels and removes.
| Tracks first open → first row | 38 ms | 26 ms | 1 255 ms | - **A finished download is moved to `<downloads>/<Subdirectory>/`**,
| Slowest first view open | 44 ms | 57 ms | 76 ms | where `Destination.Subdirectory` defaults to `${SOURCE_DIRECTORY}`
(the remote leaf folder) and is user-configurable. A taken name is
Six things worth keeping: written as `name_<ticks>.ext` (`Destination.Exists = rename`, the
default). No transfer record says where the file went.
- **The number that fell was not the number being optimised.** The - **Batch enqueue (`POST /transfers/downloads/batches`) is new in 0.26.0**
binding payload fell 20.5 → 10.45 MB (dictionary-encoded columns) and and is the only way to choose where a file lands: `options.destination`
35.9 → 12.1 MB, but what made the backend's RSS fall by 354 MB was overrides the subdirectory pattern. A batch's files land flat in it,
the *fetch not happening*: `GetTracks` was the only thing in the app so one batch per disc. On an older daemon that path is routed to the
that allocated 170 MB transiently (`sqlcgen.GetTracks` 80 MB, `json/v2` per-user enqueue as username "batches" and the object body is
128 MB, `slices.Grow` 77 MB, `bytes.Clone` 47 MB of 484 MB total rejected with 400 — which is why 400 means "no batches" here.
`alloc_space`). Encoding smaller would not have touched that. - **`GET .../downloads/{user}/{id}/position` asks the peer** and returns
- **A `dictionary` is what makes dropping fields the wrong trade.** The a bare integer. slskd's own comment on `PlaceInQueue` is "may be
first version of the column table left out `LastPlayed` and the three wildly innacurate to the point of uselessness", which is why #275 acts
larger cover tiers, and then a "select all → edit tags" over 50 000 only on two readings in a row.
tracks had to fetch every track back to open the dialog: 3 071 ms
against 88 ms before. Interning means the repeated strings are nearly
free, so the fields belong in the table; what fixed it in the end was
neither — the Tracks view hands the dialog the rows it already has.
**A payload optimisation that removes a field is a new fetch waiting
to be written.**
- **`mmap_size` is a bound, not an allocation.** #283 read as "64 MB of
mapping per connection", and the RSS of the database mapping is 46 MB
whether the bound is 64 MB or 16 MB, because a mapping costs what the
working set touches. Quartering it moved no query either (1 172 →
1 197 ms for the whole track list, 106 → 124 ms for the album list, 6
→ 8 ms for an FTS search — all inside the run-to-run spread). Kept for
the phone, where the bound is the address space the low-memory killer
reads.
- **A view that is first paint is a view that is active.** `index.html`
renders a `<track-list>`, so the track list was connected at launch
and fetched 12 MB whoever was looking at — and the fix was not in the
component but in the shell: markup is not a decision about which view
the launch lands on, so the seed is `view-hidden` and the first
navigation is what activates it. Hover and keyboard focus on a nav
item prefetch, which is the ~100 ms before the click.
- **The remaining second is the transport, not the data.** A cold open
of Tracks on 50 000 tracks is 1 255 ms, and a *raw* binding call for
the same table is 1 118–1 382 ms: none of it is the TypeScript decode
or the render. It is the Go-side query, the column encode and Wails
encoding every result twice — once for a debug log that is off
(#286). Splitting that number is what says where to go next, and the
answer was not where the payload work had been.
- **A test named for the behaviour caught the thing the design missed.**
The source sweep that pins "the whole-library track array has one
reader" was written to catch the four call sites #279 had already
converted; it found `smart-playlist-editor`, which built its value
suggestions from the same array and would have gone silently empty
once nothing loaded it. The suggestion box is a backend query now
(`SuggestSmartPlaylistValues`).
And two things about this machine, since they cost a cycle each:
- **`make ui-test` is not reliable at load average 14.** Under the
workstation's own background services, the browser provider's module
fetches fail in a different handful of files each run ("Failed to
import test file", "Cannot connect to the iframe") while every test
that runs passes. `--maxWorkers=1 --retry=2` reduces it; individual
files always pass. A failure list that changes between runs is the
environment, not the branch.
- **A measurement run inherits the machine's mood.** The first
after-#281 numbers said view opens had doubled (albums 27 → 78 ms,
settings 44 → 179 ms, Tracks first row 38 → 105 ms). Re-running
unchanged gave 26 ms and 57 ms. The tell was the same one
`NOTES.md` already records twice: before and after suspiciously
equal, or suspiciously worse, across *unrelated* measurements.
+3 -13
View File
@@ -54,19 +54,9 @@ type DB struct {
// form is silently ignored, which is why WAL was never actually on. // form is silently ignored, which is why WAL was never actually on.
const ( const (
writeDSNParams = "?_pragma=busy_timeout(5000)&_pragma=journal_mode(WAL)" writeDSNParams = "?_pragma=busy_timeout(5000)&_pragma=journal_mode(WAL)"
// The read pool's own page cache and mapping bound, sized by
// measurement rather than by appetite (#283). On a 50 000-track
// library, halving the cache and quartering the mapping moved no
// query — 1 172 → 1 197 ms for the whole track list, 106 → 124 ms
// for the album list, 6 → 8 ms for an FTS search, all inside the
// run-to-run spread — and the RSS of the database mapping was 46 MB
// under both settings, because a mapping's cost is what the working
// set touches, not the bound. The bound is what matters on a
// phone, where five connections' worth of address space is the
// thing the low-memory killer reads (cf. #52).
readDSNParams = "?_pragma=busy_timeout(5000)&_pragma=journal_mode(WAL)" + readDSNParams = "?_pragma=busy_timeout(5000)&_pragma=journal_mode(WAL)" +
"&_pragma=query_only(true)&_pragma=synchronous(NORMAL)" + "&_pragma=query_only(true)&_pragma=synchronous(NORMAL)" +
"&_pragma=cache_size(-2000)&_pragma=mmap_size(16777216)" "&_pragma=cache_size(-8000)&_pragma=mmap_size(67108864)"
// readPoolConns bounds concurrent read connections. A handful is // readPoolConns bounds concurrent read connections. A handful is
// plenty for interactive search + art/lookup fan-out and keeps WAL // plenty for interactive search + art/lookup fan-out and keeps WAL
// reader overhead small. // reader overhead small.
@@ -364,8 +354,8 @@ func applyPRAGMAs(ctx context.Context, db *sql.DB) error {
pragmas := []string{ pragmas := []string{
"PRAGMA foreign_keys = ON", "PRAGMA foreign_keys = ON",
"PRAGMA synchronous = NORMAL", "PRAGMA synchronous = NORMAL",
"PRAGMA cache_size = -2000", "PRAGMA cache_size = -8000",
"PRAGMA mmap_size = 16777216", "PRAGMA mmap_size = 67108864",
} }
for _, pragma := range pragmas { for _, pragma := range pragmas {
@@ -117,9 +117,6 @@ WHERE library_id = COALESCE(NULLIF(CAST(sqlc.arg(library_id) AS INTEGER), 0), li
-- name: GetTrackByPath :one -- name: GetTrackByPath :one
SELECT * FROM track_metadata WHERE file_path = ? LIMIT 1; SELECT * FROM track_metadata WHERE file_path = ? LIMIT 1;
-- name: GetTracksByPaths :many
SELECT * FROM track_metadata WHERE file_path IN (sqlc.slice('paths'));
-- name: GetTracksByAlbum :many -- name: GetTracksByAlbum :many
SELECT * FROM track_metadata SELECT * FROM track_metadata
WHERE album_id = sqlc.arg(album_id) WHERE album_id = sqlc.arg(album_id)
@@ -830,71 +830,6 @@ func (q *Queries) GetTracksByGenre(ctx context.Context, arg GetTracksByGenrePara
return items, nil return items, nil
} }
const getTracksByPaths = `-- name: GetTracksByPaths :many
SELECT id, file_path, length_milliseconds, title, artist_name, track_number, disc_number, album, genre, year, release_year, composer, file_type, sample_rate, bit_depth, channels, bitrate, file_size, library_id, play_count, last_played, cover_art_path, artist_mbid, release_group_mbid, recording_mbid, album_id, artist_id FROM track_metadata WHERE file_path IN (/*SLICE:paths*/?)
`
func (q *Queries) GetTracksByPaths(ctx context.Context, paths []string) ([]TrackMetadatum, error) {
query := getTracksByPaths
var queryParams []interface{}
if len(paths) > 0 {
for _, v := range paths {
queryParams = append(queryParams, v)
}
query = strings.Replace(query, "/*SLICE:paths*/?", strings.Repeat(",?", len(paths))[1:], 1)
} else {
query = strings.Replace(query, "/*SLICE:paths*/?", "NULL", 1)
}
rows, err := q.db.QueryContext(ctx, query, queryParams...)
if err != nil {
return nil, err
}
defer rows.Close()
var items []TrackMetadatum
for rows.Next() {
var i TrackMetadatum
if err := rows.Scan(
&i.ID,
&i.FilePath,
&i.LengthMilliseconds,
&i.Title,
&i.ArtistName,
&i.TrackNumber,
&i.DiscNumber,
&i.Album,
&i.Genre,
&i.Year,
&i.ReleaseYear,
&i.Composer,
&i.FileType,
&i.SampleRate,
&i.BitDepth,
&i.Channels,
&i.Bitrate,
&i.FileSize,
&i.LibraryID,
&i.PlayCount,
&i.LastPlayed,
&i.CoverArtPath,
&i.ArtistMbid,
&i.ReleaseGroupMbid,
&i.RecordingMbid,
&i.AlbumID,
&i.ArtistID,
); err != nil {
return nil, err
}
items = append(items, i)
}
if err := rows.Close(); err != nil {
return nil, err
}
if err := rows.Err(); err != nil {
return nil, err
}
return items, nil
}
const lookupTrackMetaByPaths = `-- name: LookupTrackMetaByPaths :many const lookupTrackMetaByPaths = `-- name: LookupTrackMetaByPaths :many
SELECT id, file_path, title, artist_name, album, cover_art_path, SELECT id, file_path, title, artist_name, album, cover_art_path,
artist_mbid, release_group_mbid, recording_mbid artist_mbid, release_group_mbid, recording_mbid
+26 -6
View File
@@ -4,6 +4,7 @@ import (
"bytes" "bytes"
"context" "context"
"encoding/json" "encoding/json"
"errors"
"fmt" "fmt"
"io" "io"
"net/http" "net/http"
@@ -144,17 +145,36 @@ func (c *apiClient) checkStatus(resp *http.Response) error {
case resp.StatusCode >= 400: case resp.StatusCode >= 400:
snippet, _ := io.ReadAll(io.LimitReader(resp.Body, 512)) snippet, _ := io.ReadAll(io.LimitReader(resp.Body, 512))
return fmt.Errorf( return fmt.Errorf("%w: %w", c.errUnreachable, &httpStatusError{
"%w: HTTP %d: %s", code: resp.StatusCode,
c.errUnreachable, body: strings.TrimSpace(string(snippet)),
resp.StatusCode, })
strings.TrimSpace(string(snippet)),
)
default: default:
return nil return nil
} }
} }
// httpStatusError is a non-2xx answer, kept typed so a caller can tell
// a missing endpoint from a daemon that is down.
type httpStatusError struct {
code int
body string
}
func (e *httpStatusError) Error() string {
return fmt.Sprintf("HTTP %d: %s", e.code, e.body)
}
// statusCode returns the HTTP status an error carries, or 0.
func statusCode(err error) int {
var se *httpStatusError
if errors.As(err, &se) {
return se.code
}
return 0
}
// decodeJSON decodes a JSON string into out. Providers whose auth or // decodeJSON decodes a JSON string into out. Providers whose auth or
// response handling does not fit apiClient still parse bodies the same // response handling does not fit apiClient still parse bodies the same
// way, so the helper lives here rather than being repeated. // way, so the helper lives here rather than being repeated.
+1 -1
View File
@@ -159,7 +159,7 @@ func TestSlskdCollectKeepsDiscFolders(t *testing.T) {
got, err := s.collect(Candidate{Files: []CandidateFile{ got, err := s.collect(Candidate{Files: []CandidateFile{
{Path: `\m\Album\CD1\01 Intro.flac`, IsAudio: true}, {Path: `\m\Album\CD1\01 Intro.flac`, IsAudio: true},
{Path: `\m\Album\CD2\01 Intro.flac`, IsAudio: true}, {Path: `\m\Album\CD2\01 Intro.flac`, IsAudio: true},
}}, dst) }}, dst, "", nil)
if err != nil { if err != nil {
t.Fatalf("collect: %v", err) t.Fatalf("collect: %v", err)
} }
+190
View File
@@ -0,0 +1,190 @@
package download
import (
"cmp"
"context"
"fmt"
"strings"
"yellowjacket/backend/jobs"
)
// Filling in an almost-complete album (#276).
//
// A grab that delivers nine of twelve tracks clears the completeness
// floor and is imported, and before this the other three were never
// looked for. On Soulseek that is the commonest way an album ends up
// almost right: one peer's folder is missing a track, or one file
// failed. So after an import, each missing track is searched for on
// its own and fetched from somewhere else, into the same album.
// maxFillInTracks bounds how many tracks are fetched one by one. An
// album missing more than a few is a different candidate's job, not a
// dozen single-track grabs.
const maxFillInTracks = 3
// fillIn fetches the tracks a successful import did not deliver. It
// never fails the download: the album is already imported, and a track
// it cannot find is logged and left.
func (m *Manager) fillIn(
ctx context.Context,
dl Download,
main Candidate,
imported ImportResult,
job *jobs.Handle,
) {
missing := missingTracks(dl, imported.Matched)
if len(missing) == 0 {
return
}
for _, t := range missing {
if ctx.Err() != nil {
return
}
paths, err := m.fillInTrack(ctx, dl, main, t, job)
if err != nil {
m.logger.Info(
"could not fill in a missing track",
"download", dl.ID,
"track", t.Title,
"error", err,
)
if job != nil {
job.Logf(jobs.LevelWarn, fmt.Sprintf(
"Could not find %q elsewhere: %v", t.Title, err,
))
}
continue
}
if job != nil {
job.Logf(jobs.LevelInfo, fmt.Sprintf(
"Filled in %q from another source (%d file)", t.Title, len(paths),
))
}
}
}
// missingTracks is what an import left out, when filling it in is
// worth trying: an album with a tracklist, a few tracks short.
func missingTracks(dl Download, matched []ExpectedTrack) []ExpectedTrack {
// A recording request is one track; there is no album to complete.
if dl.RecordingMBID != "" || len(dl.Expected) < 2 {
return nil
}
have := make(map[trackKey]bool, len(matched))
for _, t := range matched {
have[keyOf(t)] = true
}
var missing []ExpectedTrack
for _, t := range dl.Expected {
if !have[keyOf(t)] {
missing = append(missing, t)
}
}
// A half-empty album was a poor copy, not a nearly complete one.
if len(missing) > maxFillInTracks || 2*len(missing) >= len(dl.Expected) {
return nil
}
return missing
}
// fillInTrack searches for one track and grabs the first acceptable
// copy that is not from the source that already failed to supply it.
// One attempt: a fill-in that walks a candidate list per track would
// multiply a download's grabs by the number of gaps.
func (m *Manager) fillInTrack(
ctx context.Context,
dl Download,
main Candidate,
t ExpectedTrack,
job *jobs.Handle,
) ([]string, error) {
want := trackRequest(dl, t)
ranked, err := m.Search(ctx, want)
if err != nil {
return nil, err
}
prefs := m.preferences()
for _, c := range ranked {
if ruledOutBy(c, []Candidate{main}) || !autoAcceptable(want, c, prefs) {
continue
}
// The track is being put into an album this app placed; a
// delegate would put it in its own library instead.
if plan, err := m.planTransfer(want, c); err != nil || plan.delegated() {
continue
}
narrowed, ok := narrowTo(c, t)
if !ok {
continue
}
out := m.attemptGrab(ctx, dl, narrowed, job, []ExpectedTrack{t})
if out.item.StagingDir != "" {
if err := m.staging.Release(out.item.StagingDir); err != nil {
m.logger.Warn("could not release staging dir", "error", err)
}
}
if out.err != nil {
return nil, out.err
}
if err := m.store.SetItemImported(
ctx, out.item.ID, out.imported.Paths,
); err != nil {
m.logger.Warn("could not record imported paths", "error", err)
}
return out.imported.Paths, nil
}
return nil, ErrNoCandidates
}
// trackRequest is the search for one track of an album: the track's
// artist and title as the query, the album kept so a copy from that
// album outranks the same song off a compilation, and one expected
// track so a single file is a complete answer.
func trackRequest(dl Download, t ExpectedTrack) Download {
artist := cmp.Or(t.Artist, dl.Artist)
want := dl
want.Query = strings.TrimSpace(artist + " " + t.Title)
want.Expected = []ExpectedTrack{t}
return want
}
// narrowTo trims a candidate to the one file that aligns to t, so the
// grab fetches a track rather than whatever else the folder offered.
func narrowTo(c Candidate, t ExpectedTrack) (Candidate, bool) {
aligned, _ := matchFiles(c.Files, []ExpectedTrack{t})
for _, f := range aligned {
if f.IsAudio && f.MatchedTo == t.Position {
c.Files = []CandidateFile{f}
c.TotalSize = f.Size
return c, true
}
}
return Candidate{}, false
}
+168
View File
@@ -0,0 +1,168 @@
package download
import (
"context"
"errors"
"os"
"path/filepath"
"testing"
)
// Filling in the tracks an almost-complete album is missing (#276).
func fiveTrackTitles() []string {
return append(allTitles(), "Let Down")
}
func fiveTrackDownload() Download {
dl := fourTrackDownload()
dl.Expected = append(dl.Expected, ExpectedTrack{Position: 5, Title: "Let Down"})
return dl
}
func TestMissingTracks(t *testing.T) {
t.Parallel()
dl := fiveTrackDownload()
got := func(positions ...int) []ExpectedTrack {
out := make([]ExpectedTrack, 0, len(positions))
for _, p := range positions {
out = append(out, dl.Expected[p-1])
}
return out
}
cases := []struct {
name string
dl Download
matched []ExpectedTrack
want int
}{
{"complete", dl, got(1, 2, 3, 4, 5), 0},
{"one short", dl, got(1, 2, 3, 4), 1},
{"two short", dl, got(1, 2, 3), 2},
{"half gone is a poor copy", dl, got(1, 2), 0},
{"a recording is not an album", func() Download {
d := dl
d.RecordingMBID = "rec"
return d
}(), got(1, 2, 3, 4), 0},
}
for _, tc := range cases {
if n := len(missingTracks(tc.dl, tc.matched)); n != tc.want {
t.Errorf("%s: %d missing, want %d", tc.name, n, tc.want)
}
}
big := fiveTrackDownload()
for i := 6; i <= 20; i++ {
big.Expected = append(big.Expected, ExpectedTrack{Position: i, Title: "T" + itoa(i)})
}
if n := len(missingTracks(big, big.Expected[:16])); n != 0 {
t.Errorf("four of twenty missing: %d filled in, want none past %d", n, maxFillInTracks)
}
}
// A fill-in import takes only the file that is the missing track, and
// places it in the album with the album's tags; anything else the grab
// brought is left out.
func TestImportOnlyTakesTheMissingTrack(t *testing.T) {
t.Parallel()
f := newImportFixture(t,
"05 - Let Down.flac",
"02 - Paranoid Android.flac",
)
dl := fiveTrackDownload()
got, err := f.importer.Import(
context.Background(), dl,
Result{Dir: f.dir, Files: f.files},
ImportOptions{LibraryRoot: f.root, WriteTags: true, Only: dl.Expected[4:]},
)
if err != nil {
t.Fatalf("Import: %v", err)
}
want := filepath.Join(f.root, "Radiohead", "OK Computer", "05 Let Down.flac")
if len(got.Paths) != 1 || got.Paths[0] != want {
t.Errorf("imported %q, want only %s", got.Paths, want)
}
// A file that is not the missing track is not imported at all.
g := newImportFixture(t, "02 - Paranoid Android.flac")
if _, err := g.importer.Import(
context.Background(), dl,
Result{Dir: g.dir, Files: g.files},
ImportOptions{LibraryRoot: g.root, WriteTags: true, Only: dl.Expected[4:]},
); !errors.Is(err, ErrTooIncomplete) {
t.Errorf("Import = %v, want nothing matched", err)
}
}
// The album comes from one source missing its fifth track; the fifth is
// then found on its own at another and lands in the same album.
func TestManagerFillsInAMissingTrack(t *testing.T) {
t.Parallel()
f := newManagerFixture(t)
titles := fiveTrackTitles()
album := NewFakeProvider(1, "album", Caps{CanSearch: true, CanTransport: true})
ac := candidateFor("album-cand", titles, ".flac", 30_000_000)
ac.ProviderID = 1
album.Candidates = []Candidate{ac}
for i, tt := range titles[:4] {
album.Written[trackToken(i+1)+" - "+tt+".flac"] = []byte("audio-data")
}
single := NewFakeProvider(2, "single", Caps{CanSearch: true, CanTransport: true})
sc := Candidate{
ID: "single-cand",
Protocol: ProtocolDirect,
Title: "Radiohead - OK Computer",
Artist: "Radiohead",
Files: []CandidateFile{{
Path: "Radiohead - OK Computer/05 - Let Down.flac",
Size: 30_000_000,
}},
Health: 0.5,
ProviderID: 2,
}
single.Candidates = []Candidate{sc}
single.Written["05 - Let Down.flac"] = []byte("audio-data")
f.manager.installProvider(Config{ID: 1, Priority: 90}, album)
f.manager.installProvider(Config{ID: 2, Priority: 10}, single)
dl := fiveTrackDownload()
if _, err := f.manager.Start(context.Background(), dl); err != nil {
t.Fatalf("Start: %v", err)
}
waitForDownloadState(t, f.store, dl.ID, StateComplete)
if album.GrabCallCount() != 1 || single.GrabCallCount() != 1 {
t.Errorf(
"grabs: album=%d single=%d, want 1 and 1",
album.GrabCallCount(), single.GrabCallCount(),
)
}
for i, tt := range titles {
p := filepath.Join(f.root, "Radiohead", "OK Computer", trackToken(i+1)+" "+tt+".flac")
if _, err := os.Stat(p); err != nil {
t.Errorf("track %d not in the library: %v", i+1, err)
}
}
}
+59
View File
@@ -79,6 +79,12 @@ type ImportOptions struct {
// them. Off for delegate providers, which have already imported // them. Off for delegate providers, which have already imported
// and tagged the files themselves. // and tagged the files themselves.
WriteTags bool WriteTags bool
// Only, when set, imports just the files that align to these
// tracks of the download and skips the completeness check: it is a
// fill-in for tracks an earlier grab of the same album did not
// deliver (#276), tagged and placed as part of that album.
Only []ExpectedTrack
} }
// DefaultPathTemplate is the layout used when none is configured. // DefaultPathTemplate is the layout used when none is configured.
@@ -118,6 +124,10 @@ type ImportResult struct {
// Skipped counts non-audio files left in staging (logs, cue sheets, // Skipped counts non-audio files left in staging (logs, cue sheets,
// scene .nfo files) — deliberately not imported. // scene .nfo files) — deliberately not imported.
Skipped int Skipped int
// Matched are the expected tracks an imported file was aligned to,
// which is how a caller learns what the grab did not deliver.
Matched []ExpectedTrack
} }
// Import verifies, tags and moves a completed grab into the library. // Import verifies, tags and moves a completed grab into the library.
@@ -140,14 +150,25 @@ func (i *Importer) Import(
return ImportResult{}, ErrNoAudio return ImportResult{}, ErrNoAudio
} }
if len(opts.Only) == 0 {
if err := checkCompleteness(len(audio), dl); err != nil { if err := checkCompleteness(len(audio), dl); err != nil {
return ImportResult{}, err return ImportResult{}, err
} }
}
// Align staged files to the expected tracklist so tags and // Align staged files to the expected tracklist so tags and
// filenames reflect the release, not the uploader's naming. // filenames reflect the release, not the uploader's naming.
plan := i.planFiles(audio, dl) plan := i.planFiles(audio, dl)
if len(opts.Only) > 0 {
plan = onlyTracks(plan, opts.Only)
if len(plan) == 0 {
return ImportResult{}, fmt.Errorf(
"%w: no file matched the missing track", ErrTooIncomplete,
)
}
}
out := ImportResult{ out := ImportResult{
Paths: make([]string, 0, len(plan)), Paths: make([]string, 0, len(plan)),
Skipped: skipped, Skipped: skipped,
@@ -184,11 +205,49 @@ func (i *Importer) Import(
} }
out.Paths = append(out.Paths, dest) out.Paths = append(out.Paths, dest)
if p.Matched {
out.Matched = append(out.Matched, p.Track)
}
} }
return out, nil return out, nil
} }
// trackKey identifies an expected track within a release.
type trackKey struct{ disc, position int }
func keyOf(t ExpectedTrack) trackKey {
return trackKey{disc: t.DiscNumber, position: t.Position}
}
// onlyTracks keeps the planned files aligned to one of want. A fill-in
// grab can bring more than the one file it was after — a folder where
// the title also matched a live take — and anything else would land in
// the album as a duplicate or a stranger.
func onlyTracks(plan []plannedFile, want []ExpectedTrack) []plannedFile {
keys := make(map[trackKey]bool, len(want))
for _, t := range want {
keys[keyOf(t)] = true
}
out := make([]plannedFile, 0, len(want))
seen := map[trackKey]bool{}
for _, p := range plan {
k := keyOf(p.Track)
if !p.Matched || !keys[k] || seen[k] {
continue
}
seen[k] = true
out = append(out, p)
}
return out
}
// plannedFile pairs a staged file with the expected track it matched. // plannedFile pairs a staged file with the expected track it matched.
type plannedFile struct { type plannedFile struct {
Source string Source string
+4 -1
View File
@@ -747,8 +747,9 @@ func (m *Manager) grab(
var failed []Candidate var failed []Candidate
for { for {
out := m.attemptGrab(ctx, dl, c, job) out := m.attemptGrab(ctx, dl, c, job, nil)
if out.err == nil { if out.err == nil {
m.fillIn(ctx, dl, c, out.imported, job)
m.finishGrab(ctx, dl, out.item, out.imported, job) m.finishGrab(ctx, dl, out.item, out.imported, job)
return return
@@ -836,6 +837,7 @@ func (m *Manager) attemptGrab(
dl Download, dl Download,
c Candidate, c Candidate,
job *jobs.Handle, job *jobs.Handle,
only []ExpectedTrack,
) grabOutcome { ) grabOutcome {
// Who will move the bytes is decided before any slot is taken, so // Who will move the bytes is decided before any slot is taken, so
// the transfer waits in its own provider's queue rather than in a // the transfer waits in its own provider's queue rather than in a
@@ -942,6 +944,7 @@ func (m *Manager) attemptGrab(
opts := m.importOptions() opts := m.importOptions()
opts.WriteTags = true opts.WriteTags = true
opts.Only = only
opts.LibraryRoot, err = m.library.LibraryPath(dl.LibraryID) opts.LibraryRoot, err = m.library.LibraryPath(dl.LibraryID)
if err != nil { if err != nil {
+457 -28
View File
@@ -5,13 +5,16 @@ import (
"errors" "errors"
"fmt" "fmt"
"log/slog" "log/slog"
"net/http"
"net/url" "net/url"
"os" "os"
"path" "path"
"path/filepath" "path/filepath"
"regexp" "regexp"
"slices" "slices"
"strconv"
"strings" "strings"
"sync/atomic"
"time" "time"
"github.com/google/uuid" "github.com/google/uuid"
@@ -68,6 +71,10 @@ const (
// do not get cut off by the context deadline. // do not get cut off by the context deadline.
slskdSearchWait = 20 * time.Second slskdSearchWait = 20 * time.Second
// slskdMinSearchTimeout is the smallest searchTimeout slskd accepts,
// in seconds.
slskdMinSearchTimeout = 5
// slskdTransferPoll is how often transfer state is polled. // slskdTransferPoll is how often transfer state is polled.
slskdTransferPoll = 3 * time.Second slskdTransferPoll = 3 * time.Second
@@ -88,7 +95,8 @@ const (
// it covers a peer that queues us and never starts as well as one // it covers a peer that queues us and never starts as well as one
// that starts and stops. Ten minutes is long enough for a short // that starts and stops. Ten minutes is long enough for a short
// queue ahead of us to clear and short enough that one unresponsive // queue ahead of us to clear and short enough that one unresponsive
// peer does not hold slskd's single transfer slot for an evening. // peer does not hold a transfer slot for an evening. A queue whose
// position improves restarts it (see queueWatch).
slskdStallAfter = 10 * time.Minute slskdStallAfter = 10 * time.Minute
// slskdAbsentGrace is how long a requested file may be missing from // slskdAbsentGrace is how long a requested file may be missing from
@@ -97,6 +105,23 @@ const (
// a few polls was refused. // a few polls was refused.
slskdAbsentGrace = 30 * time.Second slskdAbsentGrace = 30 * time.Second
// slskdPositionPoll is how often a peer that has queued us is asked
// where we are in its queue. Each ask is a message to the peer, so
// it is far slower than the transfer poll.
slskdPositionPoll = time.Minute
// slskdMaxQueuePosition is the queue position past which a peer is
// not worth waiting for: at a few minutes a track, fifty albums
// ahead of us is days. slskd warns the figure can be inaccurate, so
// it takes two readings in a row to act on (see awaitTransfers).
slskdMaxQueuePosition = 50
// slskdQueueCeiling is the longest a grab waits in a peer's queue
// without a byte arriving, however steadily the queue moves. An
// improving position restarts the stall clock, so without this a
// queue moving one place an hour would hold the grab all day.
slskdQueueCeiling = time.Hour
// slskdCancelTimeout bounds the cleanup that cancels abandoned // slskdCancelTimeout bounds the cleanup that cancels abandoned
// transfers. // transfers.
slskdCancelTimeout = 15 * time.Second slskdCancelTimeout = 15 * time.Second
@@ -161,6 +186,12 @@ type slskd struct {
transferPoll time.Duration transferPoll time.Duration
stallAfter time.Duration stallAfter time.Duration
absentGrace time.Duration absentGrace time.Duration
positionPoll time.Duration
queueCeiling time.Duration
// batches is whether the daemon takes batch downloads, which is
// how a grab gets a folder of its own (see enqueue).
batches atomic.Int32
} }
// newSlskd builds the provider from config. // newSlskd builds the provider from config.
@@ -217,6 +248,8 @@ func newSlskd(
transferPoll: slskdTransferPoll, transferPoll: slskdTransferPoll,
stallAfter: slskdStallAfter, stallAfter: slskdStallAfter,
absentGrace: slskdAbsentGrace, absentGrace: slskdAbsentGrace,
positionPoll: slskdPositionPoll,
queueCeiling: slskdQueueCeiling,
}, nil }, nil
} }
@@ -289,6 +322,12 @@ type slskdTransfer struct {
BytesTransferred int64 `json:"bytesTransferred"` BytesTransferred int64 `json:"bytesTransferred"`
} }
// remotelyQueued reports whether the peer has accepted the request and
// put it in its upload queue, where it waits for a slot.
func (t slskdTransfer) remotelyQueued() bool {
return strings.Contains(t.State, "Queued") && strings.Contains(t.State, "Remotely")
}
// done reports whether the transfer reached a terminal state, and // done reports whether the transfer reached a terminal state, and
// whether it succeeded. slskd reports compound states such as // whether it succeeded. slskd reports compound states such as
// "Completed, Succeeded" and "Completed, Errored". // "Completed, Succeeded" and "Completed, Errored".
@@ -431,14 +470,18 @@ func (s *slskd) searchRequest(id, text string, minFiles int) map[string]any {
maximumPeerQueueLength = 100 maximumPeerQueueLength = 100
) )
// A tenth of the wait is left for the last poll and the responses // slskd reads this in whole seconds, counted from the last response
// fetch. // rather than from the start, with a floor of 5 (#274). A tenth of
timeout := s.searchWait - s.searchWait/10 // our own wait is left for the last poll and the responses fetch.
timeout := max(
int((s.searchWait-s.searchWait/10)/time.Second),
slskdMinSearchTimeout,
)
return map[string]any{ return map[string]any{
"id": id, "id": id,
"searchText": text, "searchText": text,
"searchTimeout": timeout.Milliseconds(), "searchTimeout": timeout,
"responseLimit": responseLimit, "responseLimit": responseLimit,
"fileLimit": fileLimit, "fileLimit": fileLimit,
"filterResponses": true, "filterResponses": true,
@@ -530,8 +573,10 @@ func isVariousArtists(artist string) bool {
// usually matches one file per folder. The two-file floor that filters // usually matches one file per folder. The two-file floor that filters
// out one-file noise for an album therefore filtered out every result // out one-file noise for an album therefore filtered out every result
// for a track, and a single-track request could never be served here. // for a track, and a single-track request could never be served here.
// A request expecting one track — a recording, or the fill-in for one
// missing from an album (#276) — takes a one-file folder.
func minFilesFor(dl Download) int { func minFilesFor(dl Download) int {
if dl.RecordingMBID != "" { if dl.RecordingMBID != "" || len(dl.Expected) == 1 {
return 1 return 1
} }
@@ -736,20 +781,99 @@ func (s *slskd) Grab(
) )
} }
// Only the per-user enqueue writes into folders other grabs share;
// a batch has a folder of its own. Until the daemon has answered a
// batch either way, take the locks anyway.
if s.batches.Load() != batchesSupported {
release, err := lockSlskdFolders(ctx, s.localFolders(c))
if err != nil {
return Result{}, err
}
defer release()
}
// slskd keeps finished transfers listed until someone removes them, // slskd keeps finished transfers listed until someone removes them,
// and a transfer is matched to the request by filename. A record // and a transfer is matched to the request by filename. A record
// left by an earlier attempt at the same file from the same peer // left by an earlier attempt at the same file from the same peer
// would otherwise be read as this attempt's answer the moment the // would otherwise be read as this attempt's answer the moment the
// first poll came back — an old failure failing a transfer that has // first poll came back — an old failure failing a transfer that has
// not started. So what is already terminal is noted before enqueueing // not started. So what is already terminal is noted before enqueueing
// and ignored after. // and ignored after. The files already on disk are noted for the
release, err := lockSlskdFolders(ctx, s.localFolders(c)) // same reason (see arrivedFile).
stale := s.terminalTransferIDs(ctx, username)
existing := s.snapshotFolders(c)
dest, err := s.enqueue(ctx, username, c)
if err != nil { if err != nil {
return Result{}, err return Result{}, err
} }
defer release()
stale := s.terminalTransferIDs(ctx, username) if err := s.awaitTransfers(
ctx, username, stale, c, onProgress,
); err != nil {
s.discardDestination(dest)
return Result{}, err
}
result, err := s.collect(c, dst, dest, existing)
s.discardDestination(dest)
return result, err
}
// Whether the daemon has the batch endpoint, learned from the first
// grab that asks.
const (
batchesUnknown int32 = iota
batchesSupported
batchesUnsupported
)
// slskdDestRoot is the folder under slskd's downloads directory that
// batch destinations are made in, so everything this app asked slskd to
// write is in one place and nothing else is.
const slskdDestRoot = "yellowjacket"
// enqueue asks slskd for a candidate's files and returns the folder,
// relative to the downloads directory, they will be written to — or ""
// when slskd will choose, which is its per-user enqueue.
//
// slskd 0.26 takes a batch with an explicit destination, which is the
// only way to know for certain where a file lands. Without one it is
// `<downloads>/<remote leaf folder>/`, shared with every other download
// of a same-named folder and with the user's own, renamed with a
// `_<ticks>` suffix when a name is taken, and moved by the user's
// `Destination.Subdirectory` setting (#274).
func (s *slskd) enqueue(
ctx context.Context,
username string,
c Candidate,
) (string, error) {
if s.batches.Load() != batchesUnsupported {
dest := slskdDestRoot + "/" + uuid.NewString()
err := s.enqueueBatches(ctx, username, c, dest)
if err == nil {
s.batches.Store(batchesSupported)
return dest, nil
}
if !batchEndpointMissing(err) {
return "", err
}
// An older daemon routes this path to the per-user enqueue with
// "batches" as the username and rejects the body, which is the
// 400; 404 and 405 are a daemon that routes it nowhere.
s.batches.Store(batchesUnsupported)
s.logger.Info(
"slskd has no batch downloads; files will be found by name",
"error", err,
)
}
wanted := make([]map[string]any, 0, len(c.Files)) wanted := make([]map[string]any, 0, len(c.Files))
for _, f := range c.Files { for _, f := range c.Files {
@@ -762,16 +886,88 @@ func (s *slskd) Grab(
if err := s.client.post( if err := s.client.post(
ctx, slskdDownloadsPath(username), wanted, nil, ctx, slskdDownloadsPath(username), wanted, nil,
); err != nil { ); err != nil {
return Result{}, err return "", err
} }
if err := s.awaitTransfers( return "", nil
ctx, username, stale, c, onProgress, }
func batchEndpointMissing(err error) bool {
switch statusCode(err) {
case http.StatusBadRequest, http.StatusNotFound, http.StatusMethodNotAllowed:
return true
default:
return false
}
}
// enqueueBatches enqueues one batch per destination folder. A batch's
// files all land directly in its destination, so a multi-disc rip needs
// one per disc or disc 2's "01" is renamed out of the way of disc 1's.
func (s *slskd) enqueueBatches(
ctx context.Context,
username string,
c Candidate,
dest string,
) error {
groups := map[string][]map[string]any{}
for _, f := range c.Files {
sub := batchSubfolder(f.Path)
groups[sub] = append(groups[sub], map[string]any{
"filename": f.Path,
"size": f.Size,
})
}
subs := make([]string, 0, len(groups))
for sub := range groups {
subs = append(subs, sub)
}
slices.Sort(subs)
for _, sub := range subs {
body := map[string]any{
"username": username,
"files": groups[sub],
"options": map[string]any{"destination": path.Join(dest, sub)},
}
if err := s.client.post(
ctx, "/api/v0/transfers/downloads/batches", body, nil,
); err != nil { ); err != nil {
return Result{}, err return err
}
} }
return s.collect(c, dst) return nil
}
// batchSubfolder is where under a batch's destination a file goes: its
// disc folder, renamed to a form slskd's path sanitising leaves alone
// and ParsePath still reads a disc number from, or nothing.
func batchSubfolder(remote string) string {
norm := strings.ReplaceAll(remote, `\`, "/")
if n, ok := discFolder(path.Base(path.Dir(norm))); ok {
return "Disc " + strconv.Itoa(n)
}
return ""
}
// discardDestination removes a batch's folder once its files have been
// collected or the grab abandoned. It is this grab's own folder under
// slskdDestRoot, so nothing in it belongs to anyone else.
func (s *slskd) discardDestination(dest string) {
if dest == "" || !strings.HasPrefix(dest, slskdDestRoot+"/") {
return
}
if err := os.RemoveAll(filepath.Join(s.downloadsPath, filepath.FromSlash(dest))); err != nil {
s.logger.Debug("could not remove slskd batch folder", "dest", dest, "error", err)
}
} }
// slskdFolders serialises grabs that land in the same local folder. // slskdFolders serialises grabs that land in the same local folder.
@@ -862,11 +1058,12 @@ func (s *slskd) terminalTransferIDs(
// state, the transfer stalls, or the caller gives up. // state, the transfer stalls, or the caller gives up.
// //
// Soulseek queues are measured in hours, so there is no deadline on the // Soulseek queues are measured in hours, so there is no deadline on the
// transfer as a whole — but there is one on *progress*. slskd's // transfer as a whole — but there is one on *progress*. A peer that
// transfer limit is one, so a peer that holds us in its queue without // holds us without sending a byte is failing this download and holding
// sending a byte is not only failing this download, it is holding every // one of the daemon's few transfer slots. After stallAfter with nothing
// other Soulseek download behind it. After stallAfter with nothing // moving the peer is given up on, and the manager tries another; a peer
// moving the peer is given up on, and the manager tries another. // that has queued us is judged by its queue position as well (see
// queueWatch).
// //
// Whatever way this ends short of every file finishing, the transfers // Whatever way this ends short of every file finishing, the transfers
// still live in slskd are cancelled there. Returning without doing so // still live in slskd are cancelled there. Returning without doing so
@@ -889,6 +1086,7 @@ func (s *slskd) awaitTransfers(
lastProgress = started lastProgress = started
lastBytes int64 lastBytes int64
live []slskdTransfer live []slskdTransfer
queue queueWatch
) )
for { for {
@@ -930,6 +1128,28 @@ func (s *slskd) awaitTransfers(
lastProgress = time.Now() lastProgress = time.Now()
} }
if err := queue.observe(ctx, s, username, tally.live, &lastProgress); err != nil {
s.cancelTransfers(username, live)
// As with a stall: what already arrived goes forward.
if tally.done > 0 {
s.logger.Info("slskd queue too long; keeping what arrived", "error", err)
return nil
}
return err
}
if tally.bytes == 0 && time.Since(started) >= s.queueCeiling {
s.cancelTransfers(username, live)
return fmt.Errorf(
"%w: %s sent nothing in %s",
ErrSlskdTimeout, username, s.queueCeiling,
)
}
if onProgress != nil { if onProgress != nil {
onProgress(Progress{ onProgress(Progress{
Current: tally.bytes, Current: tally.bytes,
@@ -982,6 +1202,77 @@ func (s *slskd) awaitTransfers(
} }
} }
// queueWatch follows our place in a peer's upload queue while nothing is
// arriving (#275).
//
// No bytes for stallAfter usually means the peer has queued us, and the
// timer alone cannot tell position 2 from position 400. So a queued
// grab asks where it stands every positionPoll: a place that improved is
// progress and restarts the stall clock, and a place past
// slskdMaxQueuePosition twice running gives the peer up at once, so the
// manager moves to the next copy without waiting out the timer.
type queueWatch struct {
lastAsk time.Time
lastPlace int
far int
}
func (q *queueWatch) observe(
ctx context.Context,
s *slskd,
username string,
live []slskdTransfer,
lastProgress *time.Time,
) error {
var queued *slskdTransfer
for i := range live {
if live[i].remotelyQueued() && live[i].ID != "" {
queued = &live[i]
break
}
}
if queued == nil || time.Since(q.lastAsk) < s.positionPoll {
return nil
}
q.lastAsk = time.Now()
var place int
if err := s.client.get(
ctx, slskdDownloadsPath(username)+"/"+url.PathEscape(queued.ID)+"/position", &place,
); err != nil {
// The peer may simply not answer; the stall timer still applies.
s.logger.Debug("slskd queue position unavailable", "peer", username, "error", err)
return nil
}
if q.lastPlace > 0 && place > 0 && place < q.lastPlace {
*lastProgress = time.Now()
}
q.lastPlace = place
if place > slskdMaxQueuePosition {
q.far++
} else {
q.far = 0
}
if q.far >= 2 {
return fmt.Errorf(
"%w: %s has us at position %d in its queue",
ErrSlskdTimeout, username, place,
)
}
return nil
}
// transferTally is one poll's reading of the files a grab asked for. // transferTally is one poll's reading of the files a grab asked for.
type transferTally struct { type transferTally struct {
done, failed int done, failed int
@@ -1102,10 +1393,17 @@ func (s *slskd) transfersFor(
} }
// collect moves finished files out of slskd's download directory into // collect moves finished files out of slskd's download directory into
// staging. slskd lays them out as <downloads>/<folder>/<file>, so each // staging.
// wanted file is looked up by its base name under the folder slskd //
// derived from the remote path. // With a batch destination each file is exactly where it was asked to
func (s *slskd) collect(c Candidate, dst string) (Result, error) { // go. Without one slskd lays files out as <downloads>/<folder>/<file>,
// and arrivedFile has to tell this grab's file from whatever else has
// that name there.
func (s *slskd) collect(
c Candidate,
dst, dest string,
existing map[string]fileStamp,
) (Result, error) {
result := Result{Dir: dst, Files: make([]string, 0, len(c.Files))} result := Result{Dir: dst, Files: make([]string, 0, len(c.Files))}
for _, f := range c.Files { for _, f := range c.Files {
@@ -1113,10 +1411,28 @@ func (s *slskd) collect(c Candidate, dst string) (Result, error) {
folder := path.Base(path.Dir(norm)) folder := path.Base(path.Dir(norm))
base := path.Base(norm) base := path.Base(norm)
src := filepath.Join(s.downloadsPath, folder, base) var (
src string
info os.FileInfo
ok bool
)
info, err := os.Stat(src) if dest != "" {
if err != nil || info.Size() == 0 { src = filepath.Join(
s.downloadsPath, filepath.FromSlash(dest), batchSubfolder(f.Path), base,
)
var err error
info, err = os.Stat(src)
ok = err == nil && info.Size() > 0
} else {
src, info, ok = arrivedFile(
filepath.Join(s.downloadsPath, folder), base, existing,
)
}
if !ok {
// Not every requested file arrives; that is expected and // Not every requested file arrives; that is expected and
// handled by completeness scoring downstream. // handled by completeness scoring downstream.
continue continue
@@ -1126,7 +1442,7 @@ func (s *slskd) collect(c Candidate, dst string) (Result, error) {
// Flattened, disc 2's "01 Intro.flac" overwrites disc 1's, and // Flattened, disc 2's "01 Intro.flac" overwrites disc 1's, and
// the importer loses the folder it reads the disc number from. // the importer loses the folder it reads the disc number from.
target := filepath.Join(dst, base) target := filepath.Join(dst, base)
if _, ok := discFolder(folder); ok { if _, disc := discFolder(folder); disc {
target = filepath.Join(dst, folder, base) target = filepath.Join(dst, folder, base)
} }
@@ -1147,3 +1463,116 @@ func (s *slskd) collect(c Candidate, dst string) (Result, error) {
return result, nil return result, nil
} }
// fileStamp is enough of a file to tell whether it has been replaced.
type fileStamp struct {
size int64
modTime time.Time
}
// snapshotFolders records the files already in the folders a per-user
// enqueue will write to, so collect does not take one of them for the
// file this grab asked for. A batch writes to a new folder and needs
// none.
func (s *slskd) snapshotFolders(c Candidate) map[string]fileStamp {
if s.batches.Load() == batchesSupported {
return nil
}
out := map[string]fileStamp{}
for _, dir := range s.localFolders(c) {
entries, err := os.ReadDir(dir)
if err != nil {
continue
}
for _, e := range entries {
info, err := e.Info()
if err != nil || !info.Mode().IsRegular() {
continue
}
out[filepath.Join(dir, e.Name())] = fileStamp{
size: info.Size(),
modTime: info.ModTime(),
}
}
}
return out
}
// arrivedFile finds the file slskd wrote for base in dir.
//
// slskd's default when a name is taken is to write `name_<ticks>.ext`
// beside it, so a file with that name left by an earlier failed attempt
// — or by the user's own download of the same folder — would otherwise
// be collected while this grab's copy sat beside it under another name.
// A candidate is the name itself or a renamed form of it that was not
// already there, unchanged, before the grab enqueued; the newest wins.
// Comparing against the snapshot rather than a clock matters because
// slskd may run on another machine whose clock is not ours.
func arrivedFile(
dir, base string,
existing map[string]fileStamp,
) (string, os.FileInfo, bool) {
entries, err := os.ReadDir(dir)
if err != nil {
return "", nil, false
}
ext := filepath.Ext(base)
stem := strings.TrimSuffix(base, ext)
var (
best string
bestInfo os.FileInfo
)
for _, e := range entries {
name := e.Name()
if name != base && !isRenamedCopy(name, stem, ext) {
continue
}
info, err := e.Info()
if err != nil || !info.Mode().IsRegular() || info.Size() == 0 {
continue
}
full := filepath.Join(dir, name)
if was, ok := existing[full]; ok &&
was.size == info.Size() && was.modTime.Equal(info.ModTime()) {
continue
}
if bestInfo == nil || info.ModTime().After(bestInfo.ModTime()) {
best, bestInfo = full, info
}
}
return best, bestInfo, bestInfo != nil
}
// isRenamedCopy reports whether name is stem_<digits>ext, which is how
// slskd names a download whose name was taken.
func isRenamedCopy(name, stem, ext string) bool {
rest, ok := strings.CutPrefix(name, stem+"_")
if !ok {
return false
}
digits, ok := strings.CutSuffix(rest, ext)
if !ok || digits == "" {
return false
}
for _, r := range digits {
if r < '0' || r > '9' {
return false
}
}
return true
}
@@ -0,0 +1,130 @@
package download
import (
"context"
"os"
"path/filepath"
"slices"
"testing"
"time"
)
// TestSlskdLive runs the provider against a real slskd daemon. It is
// skipped unless YJ_SLSKD_URL, YJ_SLSKD_API_KEY and YJ_SLSKD_DOWNLOADS
// are set, and it downloads something only when YJ_SLSKD_GRAB=1 — then
// the smallest candidate the search returns, from whichever stranger
// is sharing it.
//
// Everything else here tests the provider against a stub written from
// reading slskd's source. This is where those readings are checked:
// the search options, the responses endpoint, the batch destination,
// the cancel.
//
// YJ_SLSKD_URL=http://localhost:5030 YJ_SLSKD_API_KEY=… \
// YJ_SLSKD_DOWNLOADS=/path/to/slskd/downloads YJ_SLSKD_GRAB=1 \
// go test -run TestSlskdLive -v ./backend/download/
func TestSlskdLive(t *testing.T) {
base, key, downloads := os.Getenv("YJ_SLSKD_URL"),
os.Getenv("YJ_SLSKD_API_KEY"), os.Getenv("YJ_SLSKD_DOWNLOADS")
if base == "" || key == "" || downloads == "" {
t.Skip(
"set YJ_SLSKD_URL, YJ_SLSKD_API_KEY and YJ_SLSKD_DOWNLOADS to run against a real slskd",
)
}
query := os.Getenv("YJ_SLSKD_QUERY")
if query == "" {
query = "Radiohead OK Computer"
}
p, err := newSlskd(
Config{
ID: 1, Kind: KindSlskd, Name: "live", Enabled: true,
Settings: map[string]string{"url": base, "downloadsPath": downloads},
},
func(string) (string, error) { return key, nil },
slogDiscard(),
)
if err != nil {
t.Fatalf("newSlskd: %v", err)
}
s, ok := p.(*slskd)
if !ok {
t.Fatalf("provider is %T", p)
}
ctx := context.Background()
if err := s.Check(ctx); err != nil {
t.Fatalf("Check: %v", err)
}
started := time.Now()
got, err := s.Search(ctx, Download{Query: query})
if err != nil {
t.Fatalf("Search: %v", err)
}
t.Logf(
"search %q: %d candidates in %s",
query,
len(got),
time.Since(started).Round(time.Millisecond),
)
if len(got) == 0 {
t.Fatal("no candidates; try a more common YJ_SLSKD_QUERY")
}
timed := 0
for _, c := range got {
for _, f := range c.Files {
if f.LengthMillis > 0 {
timed++
}
}
}
t.Logf("%d files carry a length", timed)
if os.Getenv("YJ_SLSKD_GRAB") != "1" {
return
}
smallest := slices.MinFunc(got, func(a, b Candidate) int {
return int(a.TotalSize - b.TotalSize)
})
t.Logf("grabbing %q from %s (%d files, %d bytes)",
smallest.Title, smallest.Origin, len(smallest.Files), smallest.TotalSize)
s.stallAfter = 3 * time.Minute
gctx, cancel := context.WithTimeout(ctx, 15*time.Minute)
defer cancel()
res, err := s.Grab(gctx, smallest, t.TempDir(), func(p Progress) {
t.Logf("%s: %d/%d bytes", p.Phase, p.Current, p.Total)
})
if err != nil {
// A stranger going offline is not a defect; what matters is
// that the transfers were cancelled, which slskd's UI shows.
t.Fatalf("Grab: %v", err)
}
t.Logf("batches: %v", s.batches.Load() == batchesSupported)
for _, f := range res.Files {
info, err := os.Stat(f)
if err != nil || info.Size() == 0 {
t.Errorf("collected %s is missing or empty: %v", f, err)
}
}
if entries, _ := os.ReadDir(filepath.Join(downloads, slskdDestRoot)); len(entries) != 0 {
t.Errorf("%d batch folders left in slskd's downloads", len(entries))
}
}
+161 -58
View File
@@ -7,7 +7,9 @@ import (
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"os" "os"
"path"
"path/filepath" "path/filepath"
"strconv"
"strings" "strings"
"sync" "sync"
"testing" "testing"
@@ -55,6 +57,21 @@ type slskdStub struct {
// noResponsesEndpoint makes /searches/{id}/responses 404, as an // noResponsesEndpoint makes /searches/{id}/responses 404, as an
// older daemon would. // older daemon would.
noResponsesEndpoint bool noResponsesEndpoint bool
// batches makes the daemon take batch downloads, as 0.26 does.
// Without it the batch endpoint answers 400, which is what an older
// daemon's per-user route does with a batch body. batchBodies
// records each batch, and delivered is written into its destination
// under downloads when it is enqueued, keyed by file base name.
batches bool
batchBodies []map[string]any
// positions is what the queue-position endpoint answers, in order;
// the last repeats. positionAsks counts the calls.
positions []int
positionAsks int
delivered map[string]string
downloads string
} }
func newSlskdStub(t *testing.T) *slskdStub { func newSlskdStub(t *testing.T) *slskdStub {
@@ -138,6 +155,29 @@ func newSlskdStub(t *testing.T) *slskdStub {
s.paths = append(s.paths, r.URL.EscapedPath()) s.paths = append(s.paths, r.URL.EscapedPath())
s.mu.Unlock() s.mu.Unlock()
if r.Method == http.MethodGet && strings.HasSuffix(r.URL.Path, "/position") {
s.mu.Lock()
idx := min(s.positionAsks, len(s.positions)-1)
s.positionAsks++
place := 0
if idx >= 0 {
place = s.positions[idx]
}
s.mu.Unlock()
writeJSON(t, w, place)
return
}
if r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/batches") {
s.enqueueBatch(t, w, r)
return
}
switch r.Method { switch r.Method {
case http.MethodPost: case http.MethodPost:
var body []map[string]any var body []map[string]any
@@ -149,6 +189,13 @@ func newSlskdStub(t *testing.T) *slskdStub {
s.mu.Lock() s.mu.Lock()
s.enqueued = body s.enqueued = body
s.posted = true s.posted = true
for _, file := range body {
name, _ := file["filename"].(string)
norm := strings.ReplaceAll(name, `\`, "/")
s.write(t, path.Base(path.Dir(norm)), path.Base(norm))
}
s.mu.Unlock() s.mu.Unlock()
w.WriteHeader(http.StatusCreated) w.WriteHeader(http.StatusCreated)
@@ -202,6 +249,89 @@ func newSlskdStub(t *testing.T) *slskdStub {
return s return s
} }
// enqueueBatch answers the batch endpoint.
func (s *slskdStub) enqueueBatch(t *testing.T, w http.ResponseWriter, r *http.Request) {
t.Helper()
s.mu.Lock()
defer s.mu.Unlock()
if !s.batches {
w.WriteHeader(http.StatusBadRequest)
return
}
var body map[string]any
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
t.Errorf("decode batch body: %v", err)
}
s.batchBodies = append(s.batchBodies, body)
s.posted = true
files, _ := body["files"].([]any)
options, _ := body["options"].(map[string]any)
dest, _ := options["destination"].(string)
for _, f := range files {
file, _ := f.(map[string]any)
s.enqueued = append(s.enqueued, file)
name, _ := file["filename"].(string)
base := path.Base(strings.ReplaceAll(name, `\`, "/"))
s.write(t, filepath.FromSlash(dest), base)
}
w.WriteHeader(http.StatusCreated)
}
// deliver names the files that arrive once enqueued.
func (s *slskdStub) deliver(names ...string) {
s.mu.Lock()
defer s.mu.Unlock()
if s.delivered == nil {
s.delivered = map[string]string{}
}
for _, n := range names {
s.delivered[n] = "audio"
}
}
// write puts a delivered file where slskd would: under dir in the
// downloads folder, renamed name_<ticks>.ext when the name is taken, as
// slskd's default Destination.Exists does. Callers hold s.mu.
func (s *slskdStub) write(t *testing.T, dir, base string) {
t.Helper()
content, ok := s.delivered[base]
if !ok {
return
}
full := filepath.Join(s.downloads, dir)
if err := os.MkdirAll(full, 0o750); err != nil {
t.Errorf("mkdir: %v", err)
}
target := filepath.Join(full, base)
if _, err := os.Stat(target); err == nil {
ext := filepath.Ext(base)
target = filepath.Join(
full,
strings.TrimSuffix(base, ext)+"_"+strconv.FormatInt(time.Now().UnixNano(), 10)+ext,
)
}
if err := os.WriteFile(target, []byte(content), 0o600); err != nil {
t.Errorf("write: %v", err)
}
}
// reject enforces API-key auth like the real daemon. // reject enforces API-key auth like the real daemon.
func (s *slskdStub) reject(w http.ResponseWriter, r *http.Request) bool { func (s *slskdStub) reject(w http.ResponseWriter, r *http.Request) bool {
s.mu.Lock() s.mu.Lock()
@@ -234,6 +364,10 @@ func newStubSlskd(t *testing.T, stub *slskdStub) (*slskd, string) {
downloads := t.TempDir() downloads := t.TempDir()
stub.mu.Lock()
stub.downloads = downloads
stub.mu.Unlock()
p, err := newSlskd( p, err := newSlskd(
Config{ Config{
ID: 1, ID: 1,
@@ -267,6 +401,8 @@ func newStubSlskd(t *testing.T, stub *slskdStub) (*slskd, string) {
// tests about stalls and absences set their own. // tests about stalls and absences set their own.
s.stallAfter = time.Minute s.stallAfter = time.Minute
s.absentGrace = time.Minute s.absentGrace = time.Minute
s.positionPoll = time.Millisecond
s.queueCeiling = time.Hour
return s, downloads return s, downloads
} }
@@ -471,24 +607,9 @@ func TestSlskdGrabCollectsFromDownloadsFolder(t *testing.T) {
}, },
} }
s, downloads := newStubSlskd(t, stub) s, _ := newStubSlskd(t, stub)
// slskd writes into <downloads>/<folder>/<file>. stub.deliver("01 Airbag.flac", "02 Paranoid Android.flac")
folder := filepath.Join(downloads, "OK Computer")
if err := os.MkdirAll(folder, 0o750); err != nil {
t.Fatalf("mkdir: %v", err)
}
for _, name := range []string{
"01 Airbag.flac",
"02 Paranoid Android.flac",
} {
if err := os.WriteFile(
filepath.Join(folder, name), []byte("audio"), 0o600,
); err != nil {
t.Fatalf("write: %v", err)
}
}
c := Candidate{ c := Candidate{
ID: "slskd:peer:OK Computer", ID: "slskd:peer:OK Computer",
@@ -547,18 +668,9 @@ func TestSlskdGrabToleratesPartialFailure(t *testing.T) {
{Filename: `\s\Album\02 B.flac`, State: "Completed, Errored"}, {Filename: `\s\Album\02 B.flac`, State: "Completed, Errored"},
}} }}
s, downloads := newStubSlskd(t, stub) s, _ := newStubSlskd(t, stub)
folder := filepath.Join(downloads, "Album") stub.deliver("01 A.flac")
if err := os.MkdirAll(folder, 0o750); err != nil {
t.Fatalf("mkdir: %v", err)
}
if err := os.WriteFile(
filepath.Join(folder, "01 A.flac"), []byte("audio"), 0o600,
); err != nil {
t.Fatalf("write: %v", err)
}
c := Candidate{ c := Candidate{
Files: []CandidateFile{ Files: []CandidateFile{
@@ -643,23 +755,12 @@ func TestSlskdRequiresConfiguration(t *testing.T) {
} }
} }
// slskdAlbum is a two-file candidate from peer, with the files slskd // slskdAlbum is a two-file candidate from peer, whose arrived files
// would have written already in place under downloads. // the stub writes where slskd would once they are enqueued.
func slskdAlbum(t *testing.T, downloads, peer string, arrived ...string) Candidate { func slskdAlbum(t *testing.T, stub *slskdStub, peer string, arrived ...string) Candidate {
t.Helper() t.Helper()
folder := filepath.Join(downloads, "Album") stub.deliver(arrived...)
if err := os.MkdirAll(folder, 0o750); err != nil {
t.Fatalf("mkdir: %v", err)
}
for _, name := range arrived {
if err := os.WriteFile(
filepath.Join(folder, name), []byte("audio"), 0o600,
); err != nil {
t.Fatalf("write: %v", err)
}
}
return Candidate{ return Candidate{
Files: []CandidateFile{ Files: []CandidateFile{
@@ -691,11 +792,11 @@ func TestSlskdGrabGivesUpOnAStalledPeer(t *testing.T) {
{ID: "t2", Filename: `\s\Album\02 B.flac`, State: "Queued, Remotely"}, {ID: "t2", Filename: `\s\Album\02 B.flac`, State: "Queued, Remotely"},
}} }}
s, downloads := newStubSlskd(t, stub) s, _ := newStubSlskd(t, stub)
s.stallAfter = 30 * time.Millisecond s.stallAfter = 30 * time.Millisecond
_, err := s.Grab( _, err := s.Grab(
context.Background(), slskdAlbum(t, downloads, "peer"), t.TempDir(), nil, context.Background(), slskdAlbum(t, stub, "peer"), t.TempDir(), nil,
) )
if !errors.Is(err, ErrSlskdTimeout) { if !errors.Is(err, ErrSlskdTimeout) {
t.Fatalf("error = %v, want ErrSlskdTimeout", err) t.Fatalf("error = %v, want ErrSlskdTimeout", err)
@@ -728,12 +829,12 @@ func TestSlskdGrabKeepsWhatArrivedBeforeAStall(t *testing.T) {
{ID: "t2", Filename: `\s\Album\02 B.flac`, State: "Queued, Remotely"}, {ID: "t2", Filename: `\s\Album\02 B.flac`, State: "Queued, Remotely"},
}} }}
s, downloads := newStubSlskd(t, stub) s, _ := newStubSlskd(t, stub)
s.stallAfter = 30 * time.Millisecond s.stallAfter = 30 * time.Millisecond
got, err := s.Grab( got, err := s.Grab(
context.Background(), context.Background(),
slskdAlbum(t, downloads, "peer", "01 A.flac"), slskdAlbum(t, stub, "peer", "01 A.flac"),
t.TempDir(), nil, t.TempDir(), nil,
) )
if err != nil { if err != nil {
@@ -779,7 +880,7 @@ func TestSlskdGrabWaitsOnATransferThatIsMoving(t *testing.T) {
}, },
}) })
s, downloads := newStubSlskd(t, stub) s, _ := newStubSlskd(t, stub)
// A hundred polls take several times the stall window; each one // A hundred polls take several times the stall window; each one
// moves a byte. The window is kept well above one poll so a // moves a byte. The window is kept well above one poll so a
// descheduled test runner does not read as a stall. // descheduled test runner does not read as a stall.
@@ -788,7 +889,7 @@ func TestSlskdGrabWaitsOnATransferThatIsMoving(t *testing.T) {
got, err := s.Grab( got, err := s.Grab(
context.Background(), context.Background(),
slskdAlbum(t, downloads, "peer", "01 A.flac", "02 B.flac"), slskdAlbum(t, stub, "peer", "01 A.flac", "02 B.flac"),
t.TempDir(), nil, t.TempDir(), nil,
) )
if err != nil { if err != nil {
@@ -814,12 +915,12 @@ func TestSlskdGrabCountsAnUnlistedFileAsFailed(t *testing.T) {
}, },
}} }}
s, downloads := newStubSlskd(t, stub) s, _ := newStubSlskd(t, stub)
s.absentGrace = 20 * time.Millisecond s.absentGrace = 20 * time.Millisecond
got, err := s.Grab( got, err := s.Grab(
context.Background(), context.Background(),
slskdAlbum(t, downloads, "peer", "01 A.flac"), slskdAlbum(t, stub, "peer", "01 A.flac"),
t.TempDir(), nil, t.TempDir(), nil,
) )
if err != nil { if err != nil {
@@ -854,9 +955,9 @@ func TestSlskdGrabIgnoresAnEarlierAttemptsRecord(t *testing.T) {
}, },
} }
s, downloads := newStubSlskd(t, stub) s, _ := newStubSlskd(t, stub)
c := slskdAlbum(t, downloads, "peer", "01 A.flac") c := slskdAlbum(t, stub, "peer", "01 A.flac")
c.Files = c.Files[:1] c.Files = c.Files[:1]
got, err := s.Grab(context.Background(), c, t.TempDir(), nil) got, err := s.Grab(context.Background(), c, t.TempDir(), nil)
@@ -880,12 +981,12 @@ func TestSlskdGrabCancelsTransfersWhenTheCallerGivesUp(t *testing.T) {
{ID: "t2", Filename: `\s\Album\02 B.flac`, State: "Queued, Remotely"}, {ID: "t2", Filename: `\s\Album\02 B.flac`, State: "Queued, Remotely"},
}} }}
s, downloads := newStubSlskd(t, stub) s, _ := newStubSlskd(t, stub)
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond) ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
defer cancel() defer cancel()
_, err := s.Grab(ctx, slskdAlbum(t, downloads, "peer"), t.TempDir(), nil) _, err := s.Grab(ctx, slskdAlbum(t, stub, "peer"), t.TempDir(), nil)
if !errors.Is(err, ErrSlskdTimeout) { if !errors.Is(err, ErrSlskdTimeout) {
t.Fatalf("error = %v, want ErrSlskdTimeout", err) t.Fatalf("error = %v, want ErrSlskdTimeout", err)
} }
@@ -912,11 +1013,11 @@ func TestSlskdEscapesTheUsername(t *testing.T) {
}, },
}} }}
s, downloads := newStubSlskd(t, stub) s, _ := newStubSlskd(t, stub)
if _, err := s.Grab( if _, err := s.Grab(
context.Background(), context.Background(),
slskdAlbum(t, downloads, "dj a/b", "01 A.flac", "02 B.flac"), slskdAlbum(t, stub, "dj a/b", "01 A.flac", "02 B.flac"),
t.TempDir(), nil, t.TempDir(), nil,
); err != nil { ); err != nil {
t.Fatalf("Grab: %v", err) t.Fatalf("Grab: %v", err)
@@ -927,7 +1028,9 @@ func TestSlskdEscapesTheUsername(t *testing.T) {
stub.mu.Unlock() stub.mu.Unlock()
for _, p := range paths { for _, p := range paths {
if p != "/api/v0/transfers/downloads/dj%20a%2Fb" { // The batch endpoint carries the name in its body.
if p != "/api/v0/transfers/downloads/dj%20a%2Fb" &&
p != "/api/v0/transfers/downloads/batches" {
t.Errorf("transfers call went to %s", p) t.Errorf("transfers call went to %s", p)
} }
} }
+328
View File
@@ -0,0 +1,328 @@
package download
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// Where slskd writes a grab's files, and how collect finds them (#274).
// slskd reads searchTimeout in whole seconds, from the last response.
func TestSlskdSearchTimeoutIsInSeconds(t *testing.T) {
t.Parallel()
cases := []struct {
wait time.Duration
want int
}{
{20 * time.Second, 18},
{200 * time.Millisecond, slskdMinSearchTimeout},
}
for _, tc := range cases {
s := &slskd{searchWait: tc.wait}
got, ok := s.searchRequest("id", "text", 2)["searchTimeout"].(int)
if !ok || got != tc.want {
t.Errorf("wait %s: searchTimeout = %v, want %d seconds", tc.wait, got, tc.want)
}
}
}
func TestIsRenamedCopy(t *testing.T) {
t.Parallel()
cases := map[string]bool{
"01 A_638912345678901234.flac": true,
"01 A.flac": false,
"01 A_.flac": false,
"01 A_v2.flac": false,
"01 A_123.mp3": false,
"01 AB_123.flac": false,
}
for name, want := range cases {
if got := isRenamedCopy(name, "01 A", ".flac"); got != want {
t.Errorf("isRenamedCopy(%q) = %v, want %v", name, got, want)
}
}
}
func succeeded(names ...string) [][]slskdTransfer {
out := make([]slskdTransfer, 0, len(names))
for i, n := range names {
out = append(out, slskdTransfer{
ID: "t" + itoa(i),
Filename: n,
State: "Completed, Succeeded",
BytesTransferred: 500,
})
}
return [][]slskdTransfer{out}
}
// On a daemon with batches, each grab writes into a folder of its own,
// collect reads from exactly there, and the folder is gone afterwards.
func TestSlskdBatchGrabUsesItsOwnFolder(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
stub.batches = true
stub.transfers = succeeded(`\s\Album\01 A.flac`, `\s\Album\02 B.flac`)
s, downloads := newStubSlskd(t, stub)
// A same-named file in the folder a per-user enqueue would use is
// someone else's, and must not be touched.
other := filepath.Join(downloads, "Album", "01 A.flac")
if err := os.MkdirAll(filepath.Dir(other), 0o750); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(other, []byte("the user's"), 0o600); err != nil {
t.Fatal(err)
}
dst := t.TempDir()
got, err := s.Grab(
context.Background(), slskdAlbum(t, stub, "peer", "01 A.flac", "02 B.flac"), dst, nil,
)
if err != nil {
t.Fatalf("Grab: %v", err)
}
if len(got.Files) != 2 {
t.Fatalf("collected %d files, want 2", len(got.Files))
}
stub.mu.Lock()
bodies := append([]map[string]any(nil), stub.batchBodies...)
stub.mu.Unlock()
if len(bodies) != 1 {
t.Fatalf("%d batches, want 1", len(bodies))
}
if bodies[0]["username"] != "peer" {
t.Errorf("batch username = %v", bodies[0]["username"])
}
dest, _ := bodies[0]["options"].(map[string]any)["destination"].(string)
if !strings.HasPrefix(dest, slskdDestRoot+"/") {
t.Errorf("destination %q is not under %s", dest, slskdDestRoot)
}
if _, err := os.Stat(filepath.Join(downloads, filepath.FromSlash(dest))); !os.IsNotExist(err) {
t.Errorf("batch folder left behind: %v", err)
}
if data, _ := os.ReadFile(other); string(data) != "the user's" {
t.Errorf("the user's own file was taken or changed: %q", data)
}
}
// A batch's files land flat in its destination, so a two-disc rip is
// two batches, one per disc, or disc 2's "01" is renamed out of the way
// of disc 1's.
func TestSlskdBatchSplitsDiscs(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
stub.batches = true
stub.transfers = succeeded(`\s\Wall\CD1\01 In.flac`, `\s\Wall\CD2\01 Hey You.flac`)
stub.deliver("01 In.flac", "01 Hey You.flac")
s, _ := newStubSlskd(t, stub)
dst := t.TempDir()
got, err := s.Grab(context.Background(), Candidate{
Files: []CandidateFile{
{Path: `\s\Wall\CD1\01 In.flac`, Size: 500, IsAudio: true},
{Path: `\s\Wall\CD2\01 Hey You.flac`, Size: 500, IsAudio: true},
},
Payload: map[string]string{"username": "peer"},
}, dst, nil)
if err != nil {
t.Fatalf("Grab: %v", err)
}
stub.mu.Lock()
n := len(stub.batchBodies)
stub.mu.Unlock()
if n != 2 {
t.Errorf("%d batches, want one per disc", n)
}
for _, want := range []string{
filepath.Join(dst, "CD1", "01 In.flac"),
filepath.Join(dst, "CD2", "01 Hey You.flac"),
} {
found := false
for _, f := range got.Files {
found = found || f == want
}
if !found {
t.Errorf("%s not collected; got %q", want, got.Files)
}
}
}
// An abandoned batch grab leaves nothing in slskd's folder either.
func TestSlskdBatchFailureDiscardsItsFolder(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
stub.batches = true
stub.transfers = [][]slskdTransfer{{
{ID: "t0", Filename: `\s\Album\01 A.flac`, State: "Completed, Errored"},
{ID: "t1", Filename: `\s\Album\02 B.flac`, State: "Completed, Errored"},
}}
s, downloads := newStubSlskd(t, stub)
// The stub delivers the file, as a partial slskd left behind would.
if _, err := s.Grab(
context.Background(), slskdAlbum(t, stub, "peer", "01 A.flac"), t.TempDir(), nil,
); err == nil {
t.Fatal("Grab succeeded with every transfer failed")
}
entries, _ := os.ReadDir(filepath.Join(downloads, slskdDestRoot))
if len(entries) != 0 {
t.Errorf("%d batch folders left behind", len(entries))
}
}
// An older daemon answers the batch endpoint with 400; the grab falls
// back to the per-user enqueue, and later grabs do not ask again.
func TestSlskdFallsBackWithoutBatches(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
stub.transfers = succeeded(`\s\Album\01 A.flac`, `\s\Album\02 B.flac`)
s, _ := newStubSlskd(t, stub)
for range 2 {
stub.mu.Lock()
stub.posted = false
stub.pollCount = 0
stub.mu.Unlock()
if _, err := s.Grab(
context.Background(), slskdAlbum(t, stub, "peer", "01 A.flac"), t.TempDir(), nil,
); err != nil {
t.Fatalf("Grab: %v", err)
}
}
stub.mu.Lock()
paths := append([]string(nil), stub.paths...)
stub.mu.Unlock()
batchCalls := 0
for _, p := range paths {
if strings.HasSuffix(p, "/batches") {
batchCalls++
}
}
if batchCalls != 1 {
t.Errorf("asked for a batch %d times, want once", batchCalls)
}
}
// Without batches, a file of the same name already in slskd's folder is
// not this grab's: slskd wrote ours beside it as name_<ticks>.ext, and
// that is the one collected.
func TestSlskdCollectsTheRenamedCopyNotTheOldFile(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
stub.transfers = succeeded(`\s\Album\01 A.flac`, `\s\Album\02 B.flac`)
s, downloads := newStubSlskd(t, stub)
old := filepath.Join(downloads, "Album", "01 A.flac")
if err := os.MkdirAll(filepath.Dir(old), 0o750); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(old, []byte("left by an earlier attempt"), 0o600); err != nil {
t.Fatal(err)
}
dst := t.TempDir()
got, err := s.Grab(
context.Background(), slskdAlbum(t, stub, "peer", "01 A.flac", "02 B.flac"), dst, nil,
)
if err != nil {
t.Fatalf("Grab: %v", err)
}
if len(got.Files) != 2 {
t.Fatalf("collected %d files, want 2", len(got.Files))
}
data, err := os.ReadFile(filepath.Join(dst, "01 A.flac"))
if err != nil || string(data) != "audio" {
t.Errorf("collected %q, want this grab's file", data)
}
if data, _ := os.ReadFile(old); string(data) != "left by an earlier attempt" {
t.Error("the file that was already there was moved")
}
}
// And when this grab's copy never arrived, the old one is not taken in
// its place.
func TestSlskdDoesNotCollectAFileThatWasAlreadyThere(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
stub.transfers = [][]slskdTransfer{
{
{ID: "t0", Filename: `\s\Album\01 A.flac`, State: "Completed, Errored"},
{
ID: "t1",
Filename: `\s\Album\02 B.flac`,
State: "Completed, Succeeded",
BytesTransferred: 500,
},
},
}
s, downloads := newStubSlskd(t, stub)
old := filepath.Join(downloads, "Album", "01 A.flac")
if err := os.MkdirAll(filepath.Dir(old), 0o750); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(old, []byte("stale"), 0o600); err != nil {
t.Fatal(err)
}
got, err := s.Grab(
context.Background(), slskdAlbum(t, stub, "peer", "02 B.flac"), t.TempDir(), nil,
)
if err != nil {
t.Fatalf("Grab: %v", err)
}
if len(got.Files) != 1 || filepath.Base(got.Files[0]) != "02 B.flac" {
t.Errorf("collected %q, want only 02 B.flac", got.Files)
}
}
+132
View File
@@ -0,0 +1,132 @@
package download
import (
"context"
"errors"
"strings"
"testing"
"time"
)
// A peer that has queued us is judged by where we are in its queue, not
// only by a timer (#275).
func queuedThen(polls int, final string) [][]slskdTransfer {
queued := []slskdTransfer{
{ID: "t0", Filename: `\s\Album\01 A.flac`, State: "Queued, Remotely"},
{ID: "t1", Filename: `\s\Album\02 B.flac`, State: "Queued, Remotely"},
}
out := make([][]slskdTransfer, 0, polls+1)
for range polls {
out = append(out, queued)
}
return append(out, []slskdTransfer{
{ID: "t0", Filename: `\s\Album\01 A.flac`, State: final, BytesTransferred: 500},
{ID: "t1", Filename: `\s\Album\02 B.flac`, State: final, BytesTransferred: 500},
})
}
func descending(from int) []int {
out := make([]int, 0, from)
for p := from; p >= 1; p-- {
out = append(out, p)
}
return out
}
// Two readings far back in the queue give the peer up at once, rather
// than after the stall timer.
func TestSlskdGivesUpOnALongQueue(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
stub.transfers = queuedThen(100_000, "Completed, Succeeded")
stub.positions = []int{400}
s, _ := newStubSlskd(t, stub)
s.stallAfter = time.Hour
started := time.Now()
_, err := s.Grab(context.Background(), slskdAlbum(t, stub, "peer"), t.TempDir(), nil)
if !errors.Is(err, ErrSlskdTimeout) || !strings.Contains(err.Error(), "position 400") {
t.Fatalf("Grab = %v, want a queue-position give-up", err)
}
if time.Since(started) > 5*time.Second {
t.Error("the give-up waited on something other than the position")
}
if len(stub.cancelledURIs()) == 0 {
t.Error("the queued transfers were not cancelled")
}
}
// One far reading is not enough: slskd says the figure can be wildly
// wrong.
func TestSlskdOneBadPositionIsNotEnough(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
stub.transfers = queuedThen(20, "Completed, Succeeded")
stub.positions = []int{400, 3}
s, _ := newStubSlskd(t, stub)
s.positionPoll = 0
if _, err := s.Grab(
context.Background(),
slskdAlbum(t, stub, "peer", "01 A.flac", "02 B.flac"),
t.TempDir(), nil,
); err != nil {
t.Fatalf("Grab: %v", err)
}
}
// A queue that is moving is progress: the grab outlives the stall timer
// while its position improves.
func TestSlskdAMovingQueueIsProgress(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
// Near enough to wait for, with more improving readings (45) than
// there are queued polls (30), so the queue outlives the stall timer
// (30 polls of at least 2 ms against 40 ms) while still improving,
// however slowly the machine runs the loop.
stub.transfers = queuedThen(30, "Completed, Succeeded")
stub.positions = descending(slskdMaxQueuePosition - 5)
s, _ := newStubSlskd(t, stub)
s.transferPoll = 2 * time.Millisecond
s.positionPoll = 0
s.stallAfter = 40 * time.Millisecond
if _, err := s.Grab(
context.Background(),
slskdAlbum(t, stub, "peer", "01 A.flac", "02 B.flac"),
t.TempDir(), nil,
); err != nil {
t.Fatalf("Grab: %v; a moving queue was treated as a stall", err)
}
}
// However steadily the queue moves, waiting in it has a ceiling.
func TestSlskdQueueHasACeiling(t *testing.T) {
t.Parallel()
stub := newSlskdStub(t)
stub.transfers = queuedThen(100_000, "Completed, Succeeded")
stub.positions = descending(100_000)
s, _ := newStubSlskd(t, stub)
s.stallAfter = time.Hour
s.queueCeiling = 50 * time.Millisecond
_, err := s.Grab(context.Background(), slskdAlbum(t, stub, "peer"), t.TempDir(), nil)
if !errors.Is(err, ErrSlskdTimeout) {
t.Fatalf("Grab = %v, want the queue ceiling", err)
}
}
-49
View File
@@ -13,7 +13,6 @@ import (
"net/http" "net/http"
"slices" "slices"
"strings" "strings"
"sync/atomic"
"time" "time"
) )
@@ -26,19 +25,6 @@ const (
// responds with a non-2xx status code. // responds with a non-2xx status code.
var ErrListenBrainzHTTP = errors.New("listenbrainz HTTP error") var ErrListenBrainzHTTP = errors.New("listenbrainz HTTP error")
// ErrListenBrainzUnauthorized is a 401: the endpoint wants a token, and
// no retry will change that.
//
// It is separate from ErrListenBrainzHTTP because it is the one
// failure that is about *this client* rather than about the thing being
// asked for — which is what makes it the one worth latching. The
// popularity endpoints answered 401 to every request on 2026-10-05, so
// a discography backfill spent one rate-limited request per artist to
// be told the same thing: 399 of them in a minute of a real library's
// backfill, each one a log line and a wasted slot in the shared
// limiter.
var ErrListenBrainzUnauthorized = errors.New("listenbrainz requires a token")
// ListenBrainzClient is a thin HTTP client for the ListenBrainz // ListenBrainzClient is a thin HTTP client for the ListenBrainz
// popularity and labs APIs. All requests are rate-limited via the // popularity and labs APIs. All requests are rate-limited via the
// shared RateLimiter and cached via the shared Cache. // shared RateLimiter and cached via the shared Cache.
@@ -53,13 +39,6 @@ type ListenBrainzClient struct {
// SetBaseURL shape — so a test that points one client at an // SetBaseURL shape — so a test that points one client at an
// httptest server does not stop being parallel-safe. // httptest server does not stop being parallel-safe.
baseURL string baseURL string
// refused latches the first 401. A token is a property of the
// installation, not of the artist being asked about, so the answer
// is the same for every later request and asking again is pure
// cost. Per client rather than global so a test can have one that
// is refused and one that is not.
refused atomic.Bool
} }
// NewListenBrainzClient creates a ListenBrainz API client. // NewListenBrainzClient creates a ListenBrainz API client.
@@ -77,14 +56,6 @@ func NewListenBrainzClient(
} }
} }
// Unauthorized reports whether this client has been refused with a 401
// during its life. A caller that is about to do a long pass of
// requests should ask before starting it: the answer will not change
// mid-pass.
func (c *ListenBrainzClient) Unauthorized() bool {
return c.refused.Load()
}
// SetBaseURL redirects this client at another host. Tests only. // SetBaseURL redirects this client at another host. Tests only.
func (c *ListenBrainzClient) SetBaseURL(url string) { func (c *ListenBrainzClient) SetBaseURL(url string) {
c.baseURL = strings.TrimSuffix(url, "/") c.baseURL = strings.TrimSuffix(url, "/")
@@ -518,11 +489,6 @@ func (c *ListenBrainzClient) doPost(
func (c *ListenBrainzClient) doRequest( func (c *ListenBrainzClient) doRequest(
ctx context.Context, method string, url string, body []byte, ctx context.Context, method string, url string, body []byte,
) ([]byte, error) { ) ([]byte, error) {
// Asked and answered, for the rest of this client's life.
if c.refused.Load() {
return nil, fmt.Errorf("%w: %s", ErrListenBrainzUnauthorized, url)
}
c.logger.Debug("listenbrainz rate limiter wait", "url", url) c.logger.Debug("listenbrainz rate limiter wait", "url", url)
if err := c.limiter.Wait(ctx); err != nil { if err := c.limiter.Wait(ctx); err != nil {
@@ -567,21 +533,6 @@ func (c *ListenBrainzClient) doRequest(
"status", resp.StatusCode, "status", resp.StatusCode,
) )
if resp.StatusCode == http.StatusUnauthorized {
// Recorded once, at warning level, because the next thing this
// client does is stop asking: a log line per artist is the
// symptom this latch exists to remove.
if c.refused.CompareAndSwap(false, true) {
c.logger.Warn("listenbrainz refused this client: "+
"popularity data needs a token, so the rest of this "+
"run will not ask for it",
"url", url,
)
}
return nil, fmt.Errorf("%w: %s", ErrListenBrainzUnauthorized, url)
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 { if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf( return nil, fmt.Errorf(
"%w: %d %s", ErrListenBrainzHTTP, resp.StatusCode, truncateBody(respBody), "%w: %d %s", ErrListenBrainzHTTP, resp.StatusCode, truncateBody(respBody),
@@ -1,90 +0,0 @@
package explore
import (
"context"
"errors"
"log/slog"
"net/http"
"net/http/httptest"
"sync/atomic"
"testing"
"yellowjacket/backend/database"
)
// #284: the popularity endpoints answered 401 to every request, so a
// backfill spent one rate-limited request per artist to be told the same
// thing — 399 of them in a minute against a real library. A token is a
// property of the installation, not of the artist, so the first refusal
// is the answer for the whole client.
func TestListenBrainzLatchesARefusal(t *testing.T) {
t.Parallel()
var requests atomic.Int64
srv := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
requests.Add(1)
w.WriteHeader(http.StatusUnauthorized)
},
))
t.Cleanup(srv.Close)
c := NewListenBrainzClient(
NewRateLimiter(), NewCache(database.NewTestDB(t), slog.Default()), slog.Default(),
)
c.SetBaseURL(srv.URL)
for i := range 5 {
_, err := c.TopRecordingsForArtist(context.Background(), "an-mbid")
if !errors.Is(err, ErrListenBrainzUnauthorized) {
t.Fatalf("call %d: err = %v, want ErrListenBrainzUnauthorized", i, err)
}
}
if got := requests.Load(); got != 1 {
t.Errorf("requests = %d, want 1: the rest of the calls are the same answer", got)
}
if !c.Unauthorized() {
t.Error("Unauthorized() = false after a 401")
}
}
// A failure that a retry could fix must not latch: the artist stays
// unmarked and the next run asks again.
func TestListenBrainzDoesNotLatchATransientFailure(t *testing.T) {
t.Parallel()
var requests atomic.Int64
srv := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
requests.Add(1)
w.WriteHeader(http.StatusInternalServerError)
},
))
t.Cleanup(srv.Close)
c := NewListenBrainzClient(
NewRateLimiter(), NewCache(database.NewTestDB(t), slog.Default()), slog.Default(),
)
c.SetBaseURL(srv.URL)
for range 3 {
_, err := c.TopRecordingsForArtist(context.Background(), "an-mbid")
if !errors.Is(err, ErrListenBrainzHTTP) {
t.Fatalf("err = %v, want ErrListenBrainzHTTP", err)
}
}
if got := requests.Load(); got != 3 {
t.Errorf("requests = %d, want 3", got)
}
if c.Unauthorized() {
t.Error("Unauthorized() = true after a 500")
}
}
-18
View File
@@ -518,13 +518,6 @@ func (si *SearchIndex) BackfillLibraryDiscographies(ctx context.Context) {
break break
} }
// A refused client is refused for every artist: the rest of
// this pass would be the same 401, once per artist (#284). The
// artists stay unmarked, so a run with a token picks them up.
if indexLB.Unauthorized() {
break
}
work <- mbid work <- mbid
} }
@@ -541,17 +534,6 @@ func (si *SearchIndex) BackfillLibraryDiscographies(ctx context.Context) {
return return
} }
if indexLB.Unauthorized() {
si.logger.Warn("discography backfill stopped early: "+
"listenbrainz refused this client, and a token is what it wants",
"artists", total, "of", len(mbids),
)
job.logf(jobs.LevelWarn, "Stopped early: ListenBrainz needs a token")
return
}
job.logf(jobs.LevelInfo, "Filled in "+strconv.Itoa(total)+" artists") job.logf(jobs.LevelInfo, "Filled in "+strconv.Itoa(total)+" artists")
si.logger.Info("discography backfill complete", "artists", total) si.logger.Info("discography backfill complete", "artists", total)
+19 -34
View File
@@ -2,6 +2,7 @@ package library
import ( import (
"database/sql" "database/sql"
"errors"
"fmt" "fmt"
"os" "os"
"path/filepath" "path/filepath"
@@ -17,6 +18,8 @@ import (
// searchTrackLimit bounds an FTS search's result set. // searchTrackLimit bounds an FTS search's result set.
const searchTrackLimit = 500 const searchTrackLimit = 500
var errNoTracksInLibrary = errors.New("no tracks in library")
// Track is one audio file with everything a list needs to draw it. // Track is one audio file with everything a list needs to draw it.
type Track struct { type Track struct {
TrackName string TrackName string
@@ -166,46 +169,28 @@ func (l *Library) GetTrackMBIDs(filePath string) TrackMBIDs {
} }
} }
// pathLookupChunk bounds the paths bound into one IN (...) query, well // GetTracks returns every track in a library, or in all of them when
// under SQLite's bind-variable limit. // libraryID is 0.
const pathLookupChunk = 500
// GetTracksByPaths returns whole tracks for the given file paths, in
// the order asked, dropping any path that is not in the library.
// //
// It is how the frontend resolves the tracks a surface is actually // The library id is a parameter rather than a second method because the
// showing (#279). Track details from the queue, a playlist or a smart // two used to be separate queries, separate bindings and a branch at
// playlist used to look the path up in the whole library's track // every call site - and the scoped form costs nothing (measured: 23 ms
// array, which had to be loaded first — so it was fetched eagerly at // against 21 ms over 26k rows).
// startup, 20.5 MB at 26k tracks, to answer questions about a handful func (l *Library) GetTracks(libraryID int64) ([]Track, error) {
// of rows. rows, err := l.db.ReadQueries.GetTracks(l.ctx, libraryID)
func (l *Library) GetTracksByPaths(paths []string) ([]Track, error) {
byPath := make(map[string]Track, len(paths))
for start := 0; start < len(paths); start += pathLookupChunk {
chunk := paths[start:min(start+pathLookupChunk, len(paths))]
rows, err := l.db.ReadQueries.GetTracksByPaths(l.ctx, chunk)
if err != nil { if err != nil {
return nil, fmt.Errorf("could not get tracks by path: %w", err) l.logger.Error("could not retrieve audio files", "error", err)
return nil, fmt.Errorf("could not get tracks: %w", err)
} }
for _, row := range rows { l.logger.Info("audio file list", "count", len(rows), "libraryID", libraryID)
byPath[row.FilePath] = trackFromRow(row)
} if len(rows) == 0 {
return nil, errNoTracksInLibrary
} }
tracks := make([]Track, 0, len(byPath)) return tracksFromRows(rows), nil
for _, path := range paths {
if t, ok := byPath[path]; ok {
tracks = append(tracks, t)
// A path asked twice is answered once.
delete(byPath, path)
}
}
return tracks, nil
} }
// SearchTracks runs the library's FTS index and returns whole tracks. // SearchTracks runs the library's FTS index and returns whole tracks.
+4 -4
View File
@@ -58,15 +58,15 @@ func TestScan_FixtureLibraryLeavesNothingBehind(t *testing.T) {
t.Skip("fixture library is empty; run make testdata") t.Skip("fixture library is empty; run make testdata")
} }
table, err := lib.GetTrackTable(0) tracks, err := lib.GetTracks(0)
if err != nil { if err != nil {
t.Fatalf("GetTrackTable: %v", err) t.Fatalf("GetTracks: %v", err)
} }
// One track per file: the projection cannot multiply rows, because // One track per file: the projection cannot multiply rows, because
// there is no join table left to multiply them. // there is no join table left to multiply them.
if int64(len(table.FilePath)) != files { if int64(len(tracks)) != files {
t.Errorf("GetTrackTable returned %d rows for %d files", len(table.FilePath), files) t.Errorf("GetTracks returned %d rows for %d files", len(tracks), files)
} }
// Nothing shared outlives what refers to it. // Nothing shared outlives what refers to it.
-64
View File
@@ -1,64 +0,0 @@
package library
import (
"fmt"
"testing"
)
// #279: the track-details openers resolve the rows a surface is showing
// by path, instead of finding them in the whole library's array. The
// answer must keep the caller's order (a batch dialog lists them as
// selected), drop what is not in the library rather than invent it, and
// survive more paths than one IN (...) can bind.
func TestGetTracksByPaths(t *testing.T) {
t.Parallel()
lib, _ := setupTestLibrary(t)
seedAlbumsAndGenres(t, lib)
got, err := lib.GetTracksByPaths([]string{
"/other/b1.mp3", "/music/missing.mp3", "/music/a1.mp3", "/other/b1.mp3",
})
if err != nil {
t.Fatalf("GetTracksByPaths: %v", err)
}
paths := make([]string, 0, len(got))
for _, tr := range got {
paths = append(paths, tr.FilePath)
}
if want := "[/other/b1.mp3 /music/a1.mp3]"; fmt.Sprint(paths) != want {
t.Fatalf("paths = %v, want %s (caller order, missing dropped, duplicate once)", paths, want)
}
// Whole tracks, not just the key: details render every field.
if got[1].TrackName != "A1" || len(got[1].Genre) != 2 {
t.Errorf("a1 = %+v, want title A1 with two genres", got[1])
}
}
func TestGetTracksByPathsSpansChunks(t *testing.T) {
t.Parallel()
lib, _ := setupTestLibrary(t)
seedAlbumsAndGenres(t, lib)
// The real path last, behind more misses than one chunk binds, so a
// loop that only ran the first chunk would return nothing.
paths := make([]string, 0, pathLookupChunk+2)
for i := range pathLookupChunk + 1 {
paths = append(paths, fmt.Sprintf("/nowhere/%d.mp3", i))
}
paths = append(paths, "/music/a2.mp3")
got, err := lib.GetTracksByPaths(paths)
if err != nil {
t.Fatalf("GetTracksByPaths: %v", err)
}
if len(got) != 1 || got[0].FilePath != "/music/a2.mp3" {
t.Fatalf("got %d tracks (%v), want only /music/a2.mp3", len(got), got)
}
}
-190
View File
@@ -1,190 +0,0 @@
package library
import (
"fmt"
"strconv"
"strings"
)
// TrackTable is every track in a library as the Tracks view uses it:
// one array per column, and every repeated string stored once (#281).
//
// GetTracks used to answer with one object per track, which at 26 138
// tracks was 20.5 MB of JSON — ~350 bytes a row of key names, four
// cover URLs identical across an album, and artist, album and genre
// strings repeated on every track of the album. Encoding it cost the
// backend ~170 MB of transient allocation and parsing it was the
// WebView's peak. Here the keys appear once, a repeated string is a
// small integer, and the columns the Tracks view does not read are not
// sent at all: LastPlayed and the three larger cover tiers belong to
// the details dialog, which fetches whole tracks by path.
//
// The projection is still trackFromRow's — each row goes through it —
// so this is an encoding of a Track, never a second description of
// one. frontend/src/utils/track-table.ts is the only decoder.
type TrackTable struct {
// Strings holds every distinct string value; a string column holds
// indexes into it. Index 0 is always "".
Strings []string `json:"strings"`
// GenreSets holds every distinct genre list, as indexes into
// Strings; Genre holds an index into it per track.
GenreSets [][]uint32 `json:"genreSets"`
FilePath []string `json:"filePath"`
TrackName []uint32 `json:"trackName"`
ArtistName []uint32 `json:"artistName"`
Album []uint32 `json:"album"`
Composer []uint32 `json:"composer"`
FileType []uint32 `json:"fileType"`
Genre []uint32 `json:"genre"`
ArtistMBID []uint32 `json:"artistMbid"`
ReleaseGroupMBID []uint32 `json:"releaseGroupMbid"`
RecordingMBID []uint32 `json:"recordingMbid"`
CoverArtSmall []uint32 `json:"coverArtSmall"`
// LengthMs is Track.TrackLength as the number it encodes.
LengthMs []int64 `json:"lengthMs"`
TrackNumber []int64 `json:"trackNumber"`
DiscNumber []int64 `json:"discNumber"`
Year []int64 `json:"year"`
SampleRate []int64 `json:"sampleRate"`
BitDepth []int64 `json:"bitDepth"`
Channels []int64 `json:"channels"`
Bitrate []int64 `json:"bitrate"`
FileSize []int64 `json:"fileSize"`
PlayCount []int64 `json:"playCount"`
}
// trackTableBuilder interns strings and genre lists while rows are
// appended.
type trackTableBuilder struct {
table TrackTable
strings map[string]uint32
genres map[string]uint32
}
func newTrackTableBuilder(capacity int) *trackTableBuilder {
b := &trackTableBuilder{
strings: map[string]uint32{"": 0},
genres: map[string]uint32{},
}
t := &b.table
t.Strings = []string{""}
t.FilePath = make([]string, 0, capacity)
for _, col := range b.stringColumns() {
*col = make([]uint32, 0, capacity)
}
for _, col := range b.intColumns() {
*col = make([]int64, 0, capacity)
}
t.Genre = make([]uint32, 0, capacity)
return b
}
// stringColumns are the interned columns, in one place so the builder
// cannot allocate one and forget to fill it.
func (b *trackTableBuilder) stringColumns() []*[]uint32 {
t := &b.table
return []*[]uint32{
&t.TrackName, &t.ArtistName, &t.Album, &t.Composer, &t.FileType,
&t.ArtistMBID, &t.ReleaseGroupMBID, &t.RecordingMBID, &t.CoverArtSmall,
}
}
func (b *trackTableBuilder) intColumns() []*[]int64 {
t := &b.table
return []*[]int64{
&t.LengthMs, &t.TrackNumber, &t.DiscNumber, &t.Year, &t.SampleRate,
&t.BitDepth, &t.Channels, &t.Bitrate, &t.FileSize, &t.PlayCount,
}
}
func (b *trackTableBuilder) intern(s string) uint32 {
if i, ok := b.strings[s]; ok {
return i
}
i := uint32(len(b.table.Strings)) //nolint:gosec // bounded by the row count
b.table.Strings = append(b.table.Strings, s)
b.strings[s] = i
return i
}
func (b *trackTableBuilder) internGenres(genres []string) uint32 {
key := strings.Join(genres, genreDelimiter)
if i, ok := b.genres[key]; ok {
return i
}
set := make([]uint32, len(genres))
for j, g := range genres {
set[j] = b.intern(g)
}
i := uint32(len(b.table.GenreSets)) //nolint:gosec // bounded by the row count
b.table.GenreSets = append(b.table.GenreSets, set)
b.genres[key] = i
return i
}
func (b *trackTableBuilder) add(tr Track) error {
lengthMs, err := strconv.ParseInt(tr.TrackLength, 10, 64)
if err != nil {
return fmt.Errorf("track %q length %q: %w", tr.FilePath, tr.TrackLength, err)
}
t := &b.table
t.FilePath = append(t.FilePath, tr.FilePath)
strs := []string{
tr.TrackName, tr.ArtistName, tr.Album, tr.Composer, tr.FileType,
tr.ArtistMBID, tr.ReleaseGroupMBID, tr.RecordingMBID, tr.CoverArtSmall,
}
for i, col := range b.stringColumns() {
*col = append(*col, b.intern(strs[i]))
}
ints := []int64{
lengthMs, tr.TrackNumber, tr.DiscNumber, tr.Year, tr.SampleRate,
tr.BitDepth, tr.Channels, tr.Bitrate, tr.FileSize, tr.PlayCount,
}
for i, col := range b.intColumns() {
*col = append(*col, ints[i])
}
t.Genre = append(t.Genre, b.internGenres(tr.Genre))
return nil
}
// GetTrackTable returns every track in a library, or in all of them
// when libraryID is 0, as a TrackTable. An empty library is an empty
// table, not an error.
func (l *Library) GetTrackTable(libraryID int64) (TrackTable, error) {
rows, err := l.db.ReadQueries.GetTracks(l.ctx, libraryID)
if err != nil {
return TrackTable{}, fmt.Errorf("could not get tracks: %w", err)
}
b := newTrackTableBuilder(len(rows))
for i := range rows {
if err := b.add(trackFromRow(rows[i])); err != nil {
return TrackTable{}, err
}
}
return b.table, nil
}
-198
View File
@@ -1,198 +0,0 @@
package library
import (
"encoding/json"
"fmt"
"strconv"
"testing"
"yellowjacket/backend/database"
)
// seedTableLibrary seeds n albums of perAlbum tracks each, with a cover
// on every album, so the table has the repetition it exists to remove.
func seedTableLibrary(t *testing.T, lib *Library, albums, perAlbum int) {
t.Helper()
for a := range albums {
for n := range perAlbum {
database.InsertTestTrack(t, lib.db, database.TestTrack{
FilePath: fmt.Sprintf(
"/music/artist-%d/album-%d/%02d - Some Track Title.flac", a%7, a, n+1,
),
Title: fmt.Sprintf("Some Track Title %d", n+1),
Artist: fmt.Sprintf("Artist %d", a%7),
ArtistMBID: fmt.Sprintf("0b7a8d2e-0000-4000-8000-%012d", a%7),
Album: fmt.Sprintf("Album Name %d", a),
AlbumMBID: fmt.Sprintf("1c7a8d2e-0000-4000-8000-%012d", a),
RecordingMBID: fmt.Sprintf("2d7a8d2e-0000-4000-8000-%012d", a*100+n),
Genres: []string{"Ambient", fmt.Sprintf("Genre %d", a%3)},
TrackNumber: int64(n + 1),
DiscNumber: 1,
Year: 2000 + int64(a%20),
LengthMs: 200_000 + int64(n),
PlayCount: int64(n),
})
}
res, err := lib.db.ExecContext(
"INSERT INTO cover_art (file_path, mime_type) VALUES (?, 'image/jpeg')",
fmt.Sprintf("/data/covers/%064d.jpg", a),
)
if err != nil {
t.Fatalf("seed cover: %v", err)
}
coverID, _ := res.LastInsertId()
if _, err := lib.db.ExecContext(
"UPDATE albums SET cover_art_id = ? WHERE name = ?",
coverID, fmt.Sprintf("Album Name %d", a),
); err != nil {
t.Fatalf("seed album cover: %v", err)
}
}
}
// decodeTable is the Go mirror of frontend/src/utils/track-table.ts,
// for asserting the encoding loses nothing it claims to carry.
func decodeTable(t *testing.T, tbl TrackTable) []Track {
t.Helper()
str := func(col []uint32, i int) string { return tbl.Strings[col[i]] }
tracks := make([]Track, len(tbl.FilePath))
for i := range tracks {
var genres []string
for _, g := range tbl.GenreSets[tbl.Genre[i]] {
genres = append(genres, tbl.Strings[g])
}
tracks[i] = Track{
FilePath: tbl.FilePath[i],
TrackName: str(tbl.TrackName, i),
ArtistName: str(tbl.ArtistName, i),
Album: str(tbl.Album, i),
Composer: str(tbl.Composer, i),
FileType: str(tbl.FileType, i),
ArtistMBID: str(tbl.ArtistMBID, i),
ReleaseGroupMBID: str(tbl.ReleaseGroupMBID, i),
RecordingMBID: str(tbl.RecordingMBID, i),
CoverArtSmall: str(tbl.CoverArtSmall, i),
Genre: genres,
TrackLength: strconv.FormatInt(tbl.LengthMs[i], 10),
TrackNumber: tbl.TrackNumber[i],
DiscNumber: tbl.DiscNumber[i],
Year: tbl.Year[i],
SampleRate: tbl.SampleRate[i],
BitDepth: tbl.BitDepth[i],
Channels: tbl.Channels[i],
Bitrate: tbl.Bitrate[i],
FileSize: tbl.FileSize[i],
PlayCount: tbl.PlayCount[i],
}
}
return tracks
}
// #281: the table is an encoding of trackFromRow's Track, minus the
// fields the Tracks view does not read. Decoding it must give back
// exactly that, row for row.
func TestTrackTableRoundTrip(t *testing.T) {
t.Parallel()
lib, _ := setupTestLibrary(t)
seedTableLibrary(t, lib, 6, 4)
tbl, err := lib.GetTrackTable(0)
if err != nil {
t.Fatalf("GetTrackTable: %v", err)
}
rows, err := lib.db.ReadQueries.GetTracks(lib.ctx, 0)
if err != nil {
t.Fatalf("GetTracks: %v", err)
}
if len(rows) != 24 || len(tbl.FilePath) != len(rows) {
t.Fatalf("table has %d rows, query %d, want 24", len(tbl.FilePath), len(rows))
}
got := decodeTable(t, tbl)
for i, row := range rows {
want := trackFromRow(row)
// Not carried: the details dialog fetches these by path.
want.LastPlayed, want.CoverArtPath, want.CoverArtMedium, want.CoverArtLarge = "", "", "", ""
if fmt.Sprintf("%+v", got[i]) != fmt.Sprintf("%+v", want) {
t.Fatalf("row %d:\n got %+v\nwant %+v", i, got[i], want)
}
}
// It was the cover, genre and name repetition that was paid for; a
// table that interned nothing would still round-trip.
if len(tbl.GenreSets) != 3 {
t.Errorf("genre sets = %d, want 3 distinct lists", len(tbl.GenreSets))
}
if a, b := tbl.CoverArtSmall[0], tbl.CoverArtSmall[1]; a != b || tbl.Strings[a] == "" {
t.Errorf("two tracks of one album hold cover indexes %d and %d", a, b)
}
}
// The point of the table, pinned: the same rows encode to a fraction of
// the object-per-track JSON they replace. A new column has to fit
// under this or raise it on purpose.
func TestTrackTableSizeBudget(t *testing.T) {
t.Parallel()
lib, _ := setupTestLibrary(t)
seedTableLibrary(t, lib, 40, 10)
tbl, err := lib.GetTrackTable(0)
if err != nil {
t.Fatalf("GetTrackTable: %v", err)
}
rows, err := lib.db.ReadQueries.GetTracks(lib.ctx, 0)
if err != nil {
t.Fatalf("GetTracks: %v", err)
}
asTable, err := json.Marshal(tbl)
if err != nil {
t.Fatal(err)
}
asObjects, err := json.Marshal(tracksFromRows(rows))
if err != nil {
t.Fatal(err)
}
perTrack := len(asTable) / len(rows)
ratio := float64(len(asTable)) / float64(len(asObjects))
t.Logf("table %d B (%d B/track), objects %d B, ratio %.2f",
len(asTable), perTrack, len(asObjects), ratio)
if ratio > 0.25 {
t.Errorf("table is %.0f%% of the object encoding, budget 25%%", ratio*100)
}
}
func TestTrackTableEmptyLibraryIsAnAnswer(t *testing.T) {
t.Parallel()
lib, _ := setupTestLibrary(t)
tbl, err := lib.GetTrackTable(0)
if err != nil {
t.Fatalf("an empty library is an empty table, not an error: %v", err)
}
if len(tbl.FilePath) != 0 || len(tbl.Strings) != 1 {
t.Errorf("empty table = %+v", tbl)
}
}
-13
View File
@@ -3125,19 +3125,6 @@ func (s *Service) EvaluateSmartPlaylist(
return tracks, nil return tracks, nil
} }
// SuggestSmartPlaylistValues returns values of field present in the
// library that contain needle, for the rule editor's value box.
func (s *Service) SuggestSmartPlaylistValues(
field, needle string,
) ([]string, error) {
values, err := smartplaylist.SuggestValues(s.db, field, needle)
if err != nil {
return nil, fmt.Errorf("suggest smart playlist values: %w", err)
}
return values, nil
}
// PreviewSmartPlaylist evaluates a rule set from raw JSON without // PreviewSmartPlaylist evaluates a rule set from raw JSON without
// requiring a saved playlist. This powers live preview in the rule // requiring a saved playlist. This powers live preview in the rule
// editor — the frontend sends rules as they are being edited and // editor — the frontend sends rules as they are being edited and
+1 -82
View File
@@ -614,12 +614,7 @@ const leanTrackQuery = `SELECT
af.file_size, af.file_size,
af.play_count, af.play_count,
COALESCE(af.last_played, '') AS last_played COALESCE(af.last_played, '') AS last_played
FROM ` + leanTrackSource FROM (
// leanTrackSource is the joined row every rule's column resolves
// against, aliased af. Shared by Evaluate and SuggestValues so a
// suggestion is always a value a rule on the same field can match.
const leanTrackSource = `(
SELECT SELECT
af.id, af.id,
af.file_path, af.file_path,
@@ -1122,79 +1117,3 @@ func splitGenres(concatenated string) []string {
return strings.Split(concatenated, genreDelimiter) return strings.Split(concatenated, genreDelimiter)
} }
// suggestLimit bounds one suggestion answer: a combobox shows a
// screenful, and the user narrows by typing.
const suggestLimit = 50
// errNoSuggestions is a field the editor does not offer values for.
var errNoSuggestions = errors.New("field has no suggestions")
// suggestFields are the fields whose values the rule editor suggests.
// Every one is in fieldMap; numeric fields other than the years are
// ranges, where a list of every value present is no help.
var suggestFields = map[string]bool{
"title": true, "artist": true, "album": true, "genre": true,
"composer": true, "file_type": true, "year": true, "release_year": true,
}
// SuggestValues returns up to suggestLimit distinct values of field
// present in the library that contain needle (case-insensitively),
// sorted.
//
// The rule editor used to build these lists from libraryStore's
// whole-library arrays, which made the editor one more reason to fetch
// every track at startup (#279) — and gave it an empty list when the
// arrays had not landed. Asking for the values that match what has
// been typed is a few hundred bytes, whatever the library's size.
func SuggestValues(db *database.DB, field, needle string) ([]string, error) {
col, ok := fieldMap[field]
if !ok || !suggestFields[field] {
return nil, fmt.Errorf("%w: %q", errNoSuggestions, field)
}
pattern := "%" + escapeLike(needle) + "%"
// SAFETY: col comes from fieldMap, never from the caller; the
// needle is a bound parameter.
query := `SELECT DISTINCT CAST(af.` + col + ` AS TEXT) AS v FROM ` +
leanTrackSource + `
WHERE v != '' AND v != '0' AND v LIKE ? ESCAPE '\'
ORDER BY v COLLATE NOCASE LIMIT ?`
if field == "genre" {
query = `SELECT name FROM genres
WHERE name != '' AND name LIKE ? ESCAPE '\'
ORDER BY name COLLATE NOCASE LIMIT ?`
}
rows, err := db.QueryContext(query, pattern, suggestLimit)
if err != nil {
return nil, fmt.Errorf("suggest %s: %w", field, err)
}
defer func() { _ = rows.Close() }()
values := make([]string, 0, suggestLimit)
for rows.Next() {
var v string
if err := rows.Scan(&v); err != nil {
return nil, fmt.Errorf("suggest %s: %w", field, err)
}
values = append(values, v)
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("suggest %s: %w", field, err)
}
return values, nil
}
// escapeLike makes needle match itself literally inside a LIKE pattern
// with ESCAPE '\'.
func escapeLike(needle string) string {
return strings.NewReplacer(`\`, `\\`, `%`, `\%`, `_`, `\_`).Replace(needle)
}
-61
View File
@@ -1,61 +0,0 @@
package smartplaylist
import (
"errors"
"slices"
"testing"
"yellowjacket/backend/database"
)
// The rule editor's value box asks the backend for what matches the
// text typed (#279), rather than building every list from the whole
// library loaded into the frontend.
func TestSuggestValues(t *testing.T) {
t.Parallel()
db := database.NewTestDB(t)
seedSmartPlaylistData(t, db)
cases := []struct {
field, needle string
want []string
}{
// Case-insensitive substring, distinct, sorted.
{"artist", "q", []string{"QOTSA", "Queen"}},
{"album", "OPERA", []string{"A Night at the Opera"}},
// Genres come from the genre table, one value per genre, not
// from a concatenated per-track column.
{"genre", "rock", []string{
"Funk Rock", "Hard Rock", "Progressive Rock", "Rock", "Stoner Rock",
}},
// Years are numbers and are suggested as their text.
{"year", "199", []string{"1990", "1991"}},
// The needle is literal: a LIKE wildcard in it matches only itself.
{"title", "%", []string{}},
{"composer", "_", []string{}},
}
for _, tc := range cases {
got, err := SuggestValues(db, tc.field, tc.needle)
if err != nil {
t.Fatalf("SuggestValues(%q, %q): %v", tc.field, tc.needle, err)
}
if !slices.Equal(got, tc.want) {
t.Errorf("SuggestValues(%q, %q) = %q, want %q", tc.field, tc.needle, got, tc.want)
}
}
}
func TestSuggestValuesRefusesFieldsItDoesNotSuggest(t *testing.T) {
t.Parallel()
db := database.NewTestDB(t)
for _, field := range []string{"duration", "file_path", "title; DROP TABLE x"} {
if _, err := SuggestValues(db, field, ""); !errors.Is(err, errNoSuggestions) {
t.Errorf("SuggestValues(%q) err = %v, want errNoSuggestions", field, err)
}
}
}
-27
View File
@@ -1,27 +0,0 @@
import { chromium } from '@playwright/test';
const b = await chromium.launch({ executablePath: '/usr/bin/chromium' });
const p = await b.newPage();
await p.addInitScript({ path: '/mnt/vault/dev/golang/yellowjacket/.playwright/init-events.js' });
await p.goto('http://localhost:34115', { waitUntil: 'load' });
await p.evaluate(() => window.__yjEvents.ready(30000));
const out = await p.evaluate(async () => {
const time = async (label, path, args, n = 5) => {
const ms = [];
for (let i = 0; i < n; i++) {
const t0 = performance.now();
await window.__yjEvents.call(path, args, 60000);
ms.push(performance.now() - t0);
}
ms.sort((a, b) => a - b);
return { label, medianMs: Math.round(ms[Math.floor(n / 2)]), all: ms.map((m) => Math.round(m)) };
};
return [
await time('GetTrackTable(0)', 'library.Library.GetTrackTable', [0], 4),
await time('GetAlbums(0)', 'library.Library.GetAlbums', [0], 4),
await time('SearchLocal(tide)', 'explore.Service.SearchLocal', ['tide'], 4),
await time('SearchLocal(rock)', 'explore.Service.SearchLocal', ['rock'], 4),
await time('SearchTracks(tide)', 'library.Library.SearchTracks', ['tide', 0], 4),
];
});
console.log(JSON.stringify(out, null, 1));
await b.close();
+10 -181
View File
@@ -42,13 +42,6 @@
* browse number below could not see this, because it * browse number below could not see this, because it
* visits Explore without ever typing in it. * visits Explore without ever typing in it.
* heap JS heap after a scripted browse, post-GC. m3. * heap JS heap after a scripted browse, post-GC. m3.
* memory what the app holds at rest after launch, before any
* view but the landing one is opened: the backend
* process's RSS and peak, its Go heap, the page's JS
* heap, and the bytes each binding returned on the way.
* Then the same after the first open of Tracks, and how
* long that open took to its first *row* — the number
* eager loading was buying (#280/#281).
* *
* Usage: * Usage:
* node e2e/perf/measure.mjs --label before * node e2e/perf/measure.mjs --label before
@@ -58,7 +51,7 @@
* Requires a running app: `make dev-headless SEED=bulk`. * Requires a running app: `make dev-headless SEED=bulk`.
*/ */
import { readFileSync, writeFileSync, mkdirSync, existsSync, readdirSync } from 'node:fs'; import { readFileSync, writeFileSync, mkdirSync, existsSync } from 'node:fs';
import { dirname, resolve } from 'node:path'; import { dirname, resolve } from 'node:path';
import { fileURLToPath } from 'node:url'; import { fileURLToPath } from 'node:url';
import { chromium } from '@playwright/test'; import { chromium } from '@playwright/test';
@@ -197,135 +190,6 @@ const SEARCH_DEBOUNCE_MS = 150;
/* -------------------------------------------------------------------- */ /* -------------------------------------------------------------------- */
/**
* The backend process's memory, from /proc and its own pprof endpoint.
*
* Linux and a dev build only (pprof is mounted by `-tags dev`); either
* missing is a null, not a failure — this is a measurement harness, and
* a number it cannot take is reported as absent rather than as zero.
*/
async function backendMemory() {
const out = { rssMB: null, peakMB: null, anonMB: null, goHeapInuseMB: null, goHeapHeldMB: null };
const pid = findBackendPid();
if (pid) {
try {
const status = readFileSync(`/proc/${pid}/status`, 'utf8');
const kb = (k) => Number(status.match(new RegExp(`^${k}:\\s+(\\d+)`, 'm'))?.[1] ?? NaN);
out.rssMB = round(kb('VmRSS') / 1024, 1);
out.peakMB = round(kb('VmHWM') / 1024, 1);
out.anonMB = round(kb('RssAnon') / 1024, 1);
} catch {
// The process went away between finding it and reading it.
}
}
try {
const text = await (await fetch('http://localhost:6060/debug/pprof/heap?debug=1')).text();
const field = (k) => Number(text.match(new RegExp(`^# ${k} = (\\d+)`, 'm'))?.[1] ?? NaN);
out.goHeapInuseMB = round(field('HeapInuse') / 1048576, 1);
// What the runtime holds from the OS: the RSS the heap accounts for.
out.goHeapHeldMB = round((field('HeapSys') - field('HeapReleased')) / 1048576, 1);
} catch {
// Not a dev build, or pprof is not listening.
}
return out;
}
/** The pid of the running `yj-dev` binary, found by name under /proc. */
function findBackendPid() {
try {
for (const entry of readdirSync('/proc')) {
if (!/^\d+$/.test(entry)) continue;
try {
if (readFileSync(`/proc/${entry}/comm`, 'utf8').trim() === 'yj-dev') return entry;
} catch {
// Raced with an exiting process.
}
}
} catch {
// No /proc: not Linux.
}
return null;
}
/** Bytes returned per binding name, largest first. */
function bytesByBinding(calls) {
const by = {};
for (const c of calls) by[c.path] = (by[c.path] ?? 0) + (c.bytes ?? 0);
return Object.fromEntries(
Object.entries(by)
.filter(([, b]) => b > 0)
.sort((a, b) => b[1] - a[1])
.map(([k, b]) => [k, round(b / 1048576, 2)]),
);
}
async function pageMemory(page, client) {
await client.send('HeapProfiler.collectGarbage');
await page.waitForTimeout(300);
const heap = await client.send('Runtime.getHeapUsage');
const dom = await client.send('Memory.getDOMCounters');
return { jsHeapMB: round(heap.usedSize / 1048576, 1), domNodes: dom.nodes };
}
async function measureMemory(page, client) {
// Settled: the landing view has loaded and any idle warming has run.
await page.waitForTimeout(5000);
const startupCalls = await page.evaluate(() => window.__yjPerf.calls);
const atRest = {
backend: await backendMemory(),
page: await pageMemory(page, client),
bytesByBindingMB: bytesByBinding(startupCalls),
totalBindingMB: round(startupCalls.reduce((n, c) => n + (c.bytes ?? 0), 0) / 1048576, 2),
};
// First open of Tracks, to its first row: what a user waits for when
// nothing was loaded ahead of them.
const since = await page.evaluate(() => performance.now());
const firstRowMs = await page.evaluate(async () => {
const t0 = performance.now();
document.dispatchEvent(new CustomEvent('navigate', { detail: { view: 'tracks' } }));
for (;;) {
const list = document.querySelector('#main-content > track-list:not(.view-hidden)');
if (list?.shadowRoot?.querySelector('[data-testid="track-row"]')) {
return Math.round(performance.now() - t0);
}
if (performance.now() - t0 > 60000) return null;
await new Promise((r) => requestAnimationFrame(r));
}
});
await page.waitForTimeout(2000);
const tracksCalls = await page.evaluate((t) => window.__yjPerf.since(t), since);
return {
atRest,
tracksOpen: {
firstRowMs,
bytesByBindingMB: bytesByBinding(tracksCalls),
backend: await backendMemory(),
page: await pageMemory(page, client),
},
};
}
/* -------------------------------------------------------------------- */
async function measureStartup(page) { async function measureStartup(page) {
const nav = await page.evaluate(() => { const nav = await page.evaluate(() => {
const n = performance.getEntriesByType('navigation')[0]; const n = performance.getEntriesByType('navigation')[0];
@@ -360,33 +224,20 @@ async function measureStartup(page) {
scriptBytesBeforePaint, scriptBytesBeforePaint,
requests: res.length, requests: res.length,
crossOriginRequests: crossOrigin.length, crossOriginRequests: crossOrigin.length,
crossOriginHosts: [...new Set(crossOrigin.map((u) => { crossOriginHosts: [...new Set(crossOrigin.map((u) => new URL(u).host))],
try {
return new URL(u).host;
} catch {
return u;
}
}))],
}; };
}); });
// "First row on screen" is the number a user experiences as startup // "First row on screen" is the number a user experiences as startup;
// *when the app lands on Tracks*; FCP fires on the chrome around an // FCP fires on the chrome around an empty list.
// empty list.
//
// The deadline is short because since #280 a landing on any other
// view leaves the track list unloaded, and this would otherwise
// spend a minute waiting for a row that is not coming. The number
// that means something either way is `memory.tracksOpen.firstRowMs`,
// which opens Tracks and waits for its first row deliberately.
const firstRowMs = await page.evaluate(async () => { const firstRowMs = await page.evaluate(async () => {
const t0 = performance.now(); const t0 = performance.now();
const deadline = t0 + 2000; const deadline = t0 + 60000;
for (;;) { for (;;) {
const list = document.querySelector('track-list'); const list = document.querySelector('track-list');
const row = list?.shadowRoot?.querySelector('[role="row"], .track-row'); const row = list?.shadowRoot?.querySelector('[role="row"], .track-row');
if (row) return Math.round(performance.now() - t0); if (row) return Math.round(performance.now() - t0 + (performance.timeOrigin ? 0 : 0));
if (performance.now() > deadline) return null; if (performance.now() > deadline) return null;
await new Promise((r) => setTimeout(r, 16)); await new Promise((r) => setTimeout(r, 16));
} }
@@ -486,8 +337,7 @@ async function measureTrackChange(page) {
await ev.call('queue.Queue.Clear', [], 10000).catch(() => {}); await ev.call('queue.Queue.Clear', [], 10000).catch(() => {});
// The columnar list (#281): only the paths are needed here. const tracks = await ev.call('library.Library.GetAllTracks', [], 60000);
const tracks = ((await ev.call('library.Library.GetTrackTable', [0], 60000))?.filePath ?? []).map((FilePath) => ({ FilePath }));
const paths = (tracks ?? []).slice(0, 4).map((t) => t.FilePath); const paths = (tracks ?? []).slice(0, 4).map((t) => t.FilePath);
if (paths.length < 2) return { error: 'library too small to measure' }; if (paths.length < 2) return { error: 'library too small to measure' };
@@ -585,8 +435,7 @@ async function measureFavouriteToggle(page) {
const ev = window.__yjEvents; const ev = window.__yjEvents;
const perf = window.__yjPerf; const perf = window.__yjPerf;
// The columnar list (#281): only the paths are needed here. const tracks = await ev.call('library.Library.GetAllTracks', [], 60000);
const tracks = ((await ev.call('library.Library.GetTrackTable', [0], 60000))?.filePath ?? []).map((FilePath) => ({ FilePath }));
const paths = (tracks ?? []).map((t) => t.FilePath); const paths = (tracks ?? []).map((t) => t.FilePath);
if (paths.length < per * count) { if (paths.length < per * count) {
return { error: `library too small: ${paths.length} tracks` }; return { error: `library too small: ${paths.length} tracks` };
@@ -822,8 +671,7 @@ async function measurePlaylistOpen(page, client) {
let pl = (existing ?? []).find((p) => (p.Name ?? p.name) === name); let pl = (existing ?? []).find((p) => (p.Name ?? p.name) === name);
if (!pl) { if (!pl) {
// The columnar list (#281): only the paths are needed here. const tracks = await ev.call('library.Library.GetAllTracks', [], 60000);
const tracks = ((await ev.call('library.Library.GetTrackTable', [0], 60000))?.filePath ?? []).map((FilePath) => ({ FilePath }));
const paths = (tracks ?? []).map((t) => t.FilePath); const paths = (tracks ?? []).map((t) => t.FilePath);
if (paths.length < n) return { error: `library too small: ${paths.length}` }; if (paths.length < n) return { error: `library too small: ${paths.length}` };
@@ -1752,8 +1600,7 @@ async function measurePlayerBarPass(page) {
// Stage a loaded track. Deliberately the same first tracks the // Stage a loaded track. Deliberately the same first tracks the
// track-change measurement already played, so this warms no cover // track-change measurement already played, so this warms no cover
// art that a later measurement counts requests for. // art that a later measurement counts requests for.
// The columnar list (#281): only the paths are needed here. const tracks = await ev.call('library.Library.GetAllTracks', [], 60000);
const tracks = ((await ev.call('library.Library.GetTrackTable', [0], 60000))?.filePath ?? []).map((FilePath) => ({ FilePath }));
const paths = (tracks ?? []).slice(0, 4).map((t) => t.FilePath); const paths = (tracks ?? []).slice(0, 4).map((t) => t.FilePath);
if (paths.length < 2) return { error: 'library too small to measure' }; if (paths.length < 2) return { error: 'library too small to measure' };
@@ -1988,12 +1835,6 @@ async function run(label) {
loadWallMs: Date.now() - t0, loadWallMs: Date.now() - t0,
}; };
// First of all: at rest means before any measurement opens a view.
console.log(' memory at rest, then Tracks first open…');
report.memory = await measureMemory(page, client);
await page.evaluate(() => document.dispatchEvent(
new CustomEvent('navigate', { detail: { view: 'home' } }),
));
console.log(' startup…'); console.log(' startup…');
report.startup = await measureStartup(page); report.startup = await measureStartup(page);
// Before anything else navigates: every view's first open has to be // Before anything else navigates: every view's first open has to be
@@ -2056,14 +1897,6 @@ async function run(label) {
const ROWS = [ const ROWS = [
['First contentful paint', (r) => fmt(r.startup.firstContentfulPaintMs, 'ms')], ['First contentful paint', (r) => fmt(r.startup.firstContentfulPaintMs, 'ms')],
['At rest: backend RSS', (r) => fmt(r.memory?.atRest.backend.rssMB, 'MB')],
['At rest: backend peak RSS', (r) => fmt(r.memory?.atRest.backend.peakMB, 'MB')],
['At rest: Go heap held', (r) => fmt(r.memory?.atRest.backend.goHeapHeldMB, 'MB')],
['At rest: JS heap', (r) => fmt(r.memory?.atRest.page.jsHeapMB, 'MB')],
['At rest: binding bytes', (r) => fmt(r.memory?.atRest.totalBindingMB, 'MB')],
['Tracks first open: first row', (r) => fmt(r.memory?.tracksOpen.firstRowMs, 'ms')],
['Tracks open: backend peak RSS', (r) => fmt(r.memory?.tracksOpen.backend.peakMB, 'MB')],
['Tracks open: JS heap', (r) => fmt(r.memory?.tracksOpen.page.jsHeapMB, 'MB')],
['First track row', (r) => fmt(r.startup.firstRowAfterLoadMs, 'ms')], ['First track row', (r) => fmt(r.startup.firstRowAfterLoadMs, 'ms')],
['JS transferred', (r) => fmt(round(r.startup.scriptBytes / 1024), 'kB')], ['JS transferred', (r) => fmt(round(r.startup.scriptBytes / 1024), 'kB')],
['JS evaluated before first paint', (r) => fmt(round((r.startup.scriptBytesBeforePaint ?? 0) / 1024), 'kB')], ['JS evaluated before first paint', (r) => fmt(round((r.startup.scriptBytesBeforePaint ?? 0) / 1024), 'kB')],
@@ -2167,11 +2000,7 @@ function compare(a, b) {
const p = resolve(OUT_DIR, `${l}.json`); const p = resolve(OUT_DIR, `${l}.json`);
if (!existsSync(p)) throw new Error(`no measurement labelled '${l}' at ${p}`); if (!existsSync(p)) throw new Error(`no measurement labelled '${l}' at ${p}`);
try {
return JSON.parse(readFileSync(p, 'utf8')); return JSON.parse(readFileSync(p, 'utf8'));
} catch (err) {
throw new Error(`measurement '${l}' at ${p} is not valid JSON: ${err.message}`);
}
}; };
console.log(table([load(a), load(b)])); console.log(table([load(a), load(b)]));
-80
View File
@@ -1,80 +0,0 @@
/**
* Every bound method the suite names by hand must exist.
*
* A binding call carries only a method id, so a spec that names a method
* the Go side no longer has fails at *runtime*, in whichever spec
* happens to call it, with `unknown bound method name` — and nothing
* before that. `library.Library.GetTracks` was replaced by
* `GetTrackTable` (#281) and four specs kept calling the old name: the
* suite reported twenty failures across transport, bottom-bar and
* reduced-motion specs, none of which mention the library list.
*
* The known names come from `support/method-ids.mjs`, which derives them
* from the generated bindings tree that `make bindings-check` keeps
* current — so this cannot disagree with what the app can answer.
*
* It asserts first that it read something: a glob that matched nothing
* would pass over an empty list.
*/
import { readFileSync, readdirSync } from 'node:fs';
import { fileURLToPath } from 'node:url';
import { dirname, join } from 'node:path';
import { test, expect } from '../support/fixtures.js';
import { methodIDs } from '../support/method-ids.mjs';
const here = dirname(fileURLToPath(import.meta.url));
/** The specs, and the harness that calls bindings on their behalf. */
const DIRS = [here, join(here, '..', 'support')];
function sources(): Array<[string, string]> {
const out: Array<[string, string]> = [];
for (const dir of DIRS) {
for (const entry of readdirSync(dir)) {
if (!entry.endsWith('.ts')) continue;
if (entry.endsWith('.spec.ts') && entry === 'binding-names.spec.ts') continue;
const path = join(dir, entry);
out.push([path, readFileSync(path, 'utf8')]);
}
}
return out;
}
/** `'library.Library.GetTrackTable'` — package, service, method. */
const NAMED_BINDING = /'([a-z][A-Za-z0-9]*\.[A-Z]\w*\.[A-Z]\w*)'/g;
/**
* Names that are deliberately not bindings.
*
* `harness.spec.ts` calls an unknown method on purpose, to assert that
* the bridge rejects with a ReferenceError naming it rather than
* hanging — that spec is the reason a bad call is loud.
*/
const DELIBERATE = new Set(['queue.Queue.Nope']);
test('every binding named by a spec exists', () => {
// `methodIDs()` is the id -> name map the recorder names calls with.
const known = new Set(methodIDs().values());
const files = sources();
// A sweep over an empty glob passes and proves nothing.
expect(files.length, 'specs and support files read').toBeGreaterThan(20);
expect(known.size, 'bound method names derived').toBeGreaterThan(100);
const unknown: string[] = [];
for (const [path, source] of files) {
for (const [, name] of source.matchAll(NAMED_BINDING)) {
if (known.has(name!) || DELIBERATE.has(name!)) continue;
unknown.push(`${name} (${path.split('/').pop()})`);
}
}
expect([...new Set(unknown)]).toEqual([]);
});
+10 -8
View File
@@ -1,10 +1,4 @@
import { import { test, expect, callBinding, NO_QUEUE_SOURCE } from '../support/fixtures.js';
test,
expect,
callBinding,
libraryTracks,
NO_QUEUE_SOURCE,
} from '../support/fixtures.js';
import type { Page } from '@playwright/test'; import type { Page } from '@playwright/test';
/** /**
@@ -45,7 +39,15 @@ const geometry = (app: Page) =>
/** Something has to be playing before the transport draws a seek bar. */ /** Something has to be playing before the transport draws a seek bar. */
async function play(app: Page): Promise<void> { async function play(app: Page): Promise<void> {
const paths = (await libraryTracks(app)).slice(0, 3).map((t) => t.FilePath); const paths = await app.evaluate(async () => {
const tracks = (await window.__yjEvents.call(
'library.Library.GetTracks',
[0],
10_000,
)) as { FilePath: string }[];
return tracks.slice(0, 3).map((t) => t.FilePath);
});
await callBinding(app, 'queue.Queue.SetQueue', [ await callBinding(app, 'queue.Queue.SetQueue', [
paths, paths,
-58
View File
@@ -1,58 +0,0 @@
/**
* #280: launching the app must not load the track list.
*
* Every collection used to be fetched at `DOMContentLoaded` and
* refetched on every invalidation, whichever view was showing. On a
* 26 138-track library the track list was 20.5 MB of that, and encoding
* it cost the backend ~170 MB of transient allocation — for someone
* looking at Home, which draws none of it. Measured on 50 000 tracks:
* 543 MB of backend RSS at rest before, 296 MB after.
*
* The negative assertion is the point, and it needs its complement: "no
* `GetTrackTable`" also holds on a build that fetches nothing at all,
* so the same spec opens Tracks and watches it arrive.
*/
import { test, expect, bindingCalls, navigateTo } from '../support/fixtures.js';
const TRACKS = 'library.Library.GetTrackTable';
/** The seed's default page, which is what a launch lands on. */
const LANDING_VIEW = 'home';
test('launching on Home does not load the track list', async ({ app }) => {
// Past the store's idle warm-up: "nothing was fetched" has to mean
// nothing, not "nothing has happened yet".
await app.waitForTimeout(3_500);
const calls = await bindingCalls(app);
expect(calls, 'the warm-up for the small collections ran').toContain(
'library.Library.GetAlbums',
);
expect(calls, `the track list was fetched at launch (${LANDING_VIEW})`).not.toContain(
TRACKS,
);
});
test('opening Tracks loads it, and only then', async ({ app }) => {
await app.waitForTimeout(1_000);
expect(await bindingCalls(app)).not.toContain(TRACKS);
await navigateTo(app, 'tracks');
await expect(app.getByTestId('track-row').first()).toBeVisible();
expect(await bindingCalls(app)).toContain(TRACKS);
});
test('a hover on the nav item starts the load before the click', async ({ app }) => {
await app.waitForTimeout(1_000);
const nav = app.getByTestId('nav-tracks');
// The pointer entering the item is the whole mechanism; nothing is
// clicked, so the data must arrive because of the hover alone.
await nav.hover();
await expect
.poll(async () => (await bindingCalls(app)).includes(TRACKS))
.toBe(true);
});
+9 -3
View File
@@ -1,7 +1,6 @@
import { import {
test, test,
expect, expect,
libraryTracks,
callBinding, callBinding,
openTheQueue, openTheQueue,
NO_QUEUE_SOURCE, NO_QUEUE_SOURCE,
@@ -57,11 +56,18 @@ async function menuLabels(app: Page): Promise<string[]> {
* the wrong "no link". * the wrong "no link".
*/ */
async function queueThree(app: Page): Promise<void> { async function queueThree(app: Page): Promise<void> {
const tracks = await libraryTracks(app); const paths = await app.evaluate(async () => {
const paths = tracks const tracks = (await window.__yjEvents.call(
'library.Library.GetTracks',
[0],
10_000,
)) as { FilePath: string; Album: string; ArtistName: string }[];
return tracks
.filter((t) => t.Album !== '' && t.ArtistName !== '') .filter((t) => t.Album !== '' && t.ArtistName !== '')
.slice(0, 3) .slice(0, 3)
.map((t) => t.FilePath); .map((t) => t.FilePath);
});
await callBinding(app, 'queue.Queue.SetQueue', [ await callBinding(app, 'queue.Queue.SetQueue', [
paths, paths,
+9 -5
View File
@@ -1,4 +1,4 @@
import { test, expect, libraryTracks } from '../support/fixtures.js'; import { test, expect } from '../support/fixtures.js';
/** /**
* Now Playing on a short screen (#51). * Now Playing on a short screen (#51).
@@ -57,15 +57,19 @@ const REFLOW_AT = 500;
/** Put a track in the player, so the view has art and names to lay out. */ /** Put a track in the player, so the view has art and names to lay out. */
async function stageATrack(page: Page): Promise<void> { async function stageATrack(page: Page): Promise<void> {
const paths = (await libraryTracks(page)).slice(0, 4).map((t) => t.FilePath); await page.evaluate(async () => {
const tracks = (await window.__yjEvents.call(
'library.Library.GetTracks',
[0],
10_000,
)) as { FilePath: string }[];
await page.evaluate(async (paths) => {
await window.__yjEvents.call( await window.__yjEvents.call(
'queue.Queue.SetQueue', 'queue.Queue.SetQueue',
[paths, 0, false, { type: '', id: 0, label: '' }], [tracks.slice(0, 4).map((t) => t.FilePath), 0, false, { type: '', id: 0, label: '' }],
10_000, 10_000,
); );
}, paths); });
} }
/** Open the full-screen view and wait for the shell to say so. */ /** Open the full-screen view and wait for the shell to say so. */
+5 -2
View File
@@ -1,5 +1,4 @@
import { import {
libraryTracks,
test, test,
expect, expect,
callBinding, callBinding,
@@ -50,7 +49,11 @@ async function rectOf(app: Page, selector: string): Promise<Rect | null> {
* three rectangles. * three rectangles.
*/ */
async function play(app: Page): Promise<void> { async function play(app: Page): Promise<void> {
const tracks = await libraryTracks(app); const tracks = await callBinding<{ FilePath: string; TrackName: string }[]>(
app,
'library.Library.GetTracks',
[0],
);
// `TrackName`, not `Title`: that is what the library model calls it. // `TrackName`, not `Title`: that is what the library model calls it.
const long = tracks.find((t) => t.TrackName === LONG_TRACK); const long = tracks.find((t) => t.TrackName === LONG_TRACK);
+11 -4
View File
@@ -1,4 +1,4 @@
import { test, expect, LONG_TRACK, libraryTracks } from '../support/fixtures.js'; import { test, expect, LONG_TRACK } from '../support/fixtures.js';
/** /**
* The phone shell (plan 016 B2, phase 1). * The phone shell (plan 016 B2, phase 1).
@@ -235,8 +235,15 @@ test.describe('the shell on a phone', () => {
// nothing in particular and picked a 2-second track, which had // nothing in particular and picked a 2-second track, which had
// finished before the assertions ran. The placeholder check below // finished before the assertions ran. The placeholder check below
// is what actually holds the property this test needs. // is what actually holds the property this test needs.
const bare = (await libraryTracks(app)).find((t) => t.TrackName === LONG_TRACK); const started = await app.evaluate(async (longTitle) => {
const started = await app.evaluate(async (bare) => { const tracks = (await window.__yjEvents.call(
'library.Library.GetTracks',
[0],
10_000,
)) as { FilePath: string; TrackName: string }[];
const bare = tracks.find((t) => t.TrackName === longTitle);
if (!bare) return null; if (!bare) return null;
await window.__yjEvents.call( await window.__yjEvents.call(
@@ -247,7 +254,7 @@ test.describe('the shell on a phone', () => {
await window.__yjEvents.call('queue.Queue.Play', [], 5_000); await window.__yjEvents.call('queue.Queue.Play', [], 5_000);
return bare.TrackName; return bare.TrackName;
}, bare); }, LONG_TRACK);
expect(started).toBe(LONG_TRACK); expect(started).toBe(LONG_TRACK);
+9 -5
View File
@@ -1,4 +1,4 @@
import { test, expect, libraryTracks } from '../support/fixtures.js'; import { test, expect } from '../support/fixtures.js';
/** /**
* The phone's transport (#59, #56). * The phone's transport (#59, #56).
@@ -64,15 +64,19 @@ async function sizeOf(
/** Put something in the queue, so the transport has a track to act on. */ /** Put something in the queue, so the transport has a track to act on. */
async function stageATrack(page: Page): Promise<void> { async function stageATrack(page: Page): Promise<void> {
const paths = (await libraryTracks(page)).slice(0, 4).map((t) => t.FilePath); await page.evaluate(async () => {
const tracks = (await window.__yjEvents.call(
'library.Library.GetTracks',
[0],
10_000,
)) as { FilePath: string }[];
await page.evaluate(async (paths) => {
await window.__yjEvents.call( await window.__yjEvents.call(
'queue.Queue.SetQueue', 'queue.Queue.SetQueue',
[paths, 0, false, { type: '', id: 0, label: '' }], [tracks.slice(0, 4).map((t) => t.FilePath), 0, false, { type: '', id: 0, label: '' }],
10_000, 10_000,
); );
}, paths); });
} }
test.describe('the phone bar carries three controls', () => { test.describe('the phone bar carries three controls', () => {
+1 -16
View File
@@ -40,21 +40,6 @@ const FINISH_TIMEOUT = 60_000;
const libraryCalls = async (app: Page): Promise<string[]> => const libraryCalls = async (app: Page): Promise<string[]> =>
(await bindingCalls(app)).filter((c) => c.startsWith('library.Library.')); (await bindingCalls(app)).filter((c) => c.startsWith('library.Library.'));
/**
* The bindings that fetch a collection.
*
* A prefix test (GetAll*) used to stand in for this, and matched
* exactly one name — GetAllLibrariesWithTrackCounts — so the assertion
* below held whatever the app refetched. The sweep in
* binding-names.spec.ts is what found it.
*/
const COLLECTION_FETCHES = [
'library.Library.GetTrackTable',
'library.Library.GetAlbums',
'library.Library.GetArtists',
'library.Library.GetGenres',
];
/** /**
* Select rows by dispatching on the row rather than clicking it. * Select rows by dispatching on the row rather than clicking it.
* *
@@ -131,7 +116,7 @@ test.describe('a finished track', () => {
const refetched = await libraryCalls(app); const refetched = await libraryCalls(app);
expect( expect(
refetched.filter((c) => COLLECTION_FETCHES.includes(c)), refetched.filter((c) => c.startsWith('library.Library.GetAll')),
'a play refetched a collection', 'a play refetched a collection',
).toEqual([]); ).toEqual([]);
+12 -2
View File
@@ -1,5 +1,4 @@
import { import {
libraryTracks,
test, test,
expect, expect,
callBinding, callBinding,
@@ -31,7 +30,18 @@ async function order(app: Page): Promise<string[]> {
} }
async function queueFourAndOpen(app: Page): Promise<string[]> { async function queueFourAndOpen(app: Page): Promise<string[]> {
const paths = (await libraryTracks(app)).slice(0, 4).map((t) => t.FilePath); const paths: string[] = await app.evaluate(async () => {
// One argument, and 0 means every library: the scoped and
// unscoped list queries collapsed into one when the schema did
// (plan 013 R3), so `GetTracks()` no longer exists to call.
const tracks = await window.__yjEvents.call(
'library.Library.GetTracks',
[0],
10_000,
);
return (tracks as { FilePath: string }[]).slice(0, 4).map((t) => t.FilePath);
});
await callBinding(app, 'queue.Queue.SetQueue', [paths, 0, false, NO_QUEUE_SOURCE]); await callBinding(app, 'queue.Queue.SetQueue', [paths, 0, false, NO_QUEUE_SOURCE]);
+8 -5
View File
@@ -1,5 +1,4 @@
import { import {
libraryTracks,
test, test,
expect, expect,
callBinding, callBinding,
@@ -87,11 +86,15 @@ const selected = (app: Page) =>
* is read back. * is read back.
*/ */
async function queueSixAndOpen(app: Page): Promise<void> { async function queueSixAndOpen(app: Page): Promise<void> {
const paths = await app.evaluate(async (longTitle) => {
// `TrackName`, not `Title`: the library model names it after the // `TrackName`, not `Title`: the library model names it after the
// tag, and the *queue* is what calls it `title`. // tag, and the *queue* is what calls it `title`.
const tracks = await libraryTracks(app); const tracks = (await window.__yjEvents.call(
const longTitle = LONG_TRACK; 'library.Library.GetTracks',
const paths = (() => { [0],
10_000,
)) as { FilePath: string; TrackName: string; Album: string }[];
const long = tracks.find((t) => t.TrackName === longTitle); const long = tracks.find((t) => t.TrackName === longTitle);
/** /**
@@ -128,7 +131,7 @@ async function queueSixAndOpen(app: Page): Promise<void> {
long!.FilePath, long!.FilePath,
...rest.slice(3).map((t) => t.FilePath), ...rest.slice(3).map((t) => t.FilePath),
]; ];
})(); }, LONG_TRACK);
await callBinding(app, 'queue.Queue.SetQueue', [ await callBinding(app, 'queue.Queue.SetQueue', [
paths, paths,
+13 -3
View File
@@ -2,7 +2,6 @@ import {
test, test,
expect, expect,
callBinding, callBinding,
libraryTracks,
waitForEvent, waitForEvent,
NO_QUEUE_SOURCE, NO_QUEUE_SOURCE,
} from '../support/fixtures.js'; } from '../support/fixtures.js';
@@ -57,9 +56,20 @@ async function playTheLongOne(app: Page): Promise<void> {
window.dispatchEvent(new CustomEvent('yj-scroll-mode-changed')); window.dispatchEvent(new CustomEvent('yj-scroll-mode-changed'));
}); });
const paths: string[] = (await libraryTracks(app)) const paths: string[] = await app.evaluate(async (needle) => {
.filter((t) => t.TrackName.startsWith(LONG_TITLE)) // One argument, and 0 means every library: the scoped and
// unscoped list queries collapsed into one when the schema did
// (plan 013 R3), so `GetTracks()` no longer exists to call.
const tracks = await window.__yjEvents.call(
'library.Library.GetTracks',
[0],
10_000,
);
return (tracks as { TrackName: string; FilePath: string }[])
.filter((t) => t.TrackName.startsWith(needle))
.map((t) => t.FilePath); .map((t) => t.FilePath);
}, LONG_TITLE);
expect(paths.length).toBeGreaterThan(0); expect(paths.length).toBeGreaterThan(0);
-33
View File
@@ -96,39 +96,6 @@ export async function callBinding<T = unknown>(
) as Promise<T>; ) as Promise<T>;
} }
/** A library track as the specs use it: the path and its names. */
export interface LibraryTrack {
FilePath: string;
TrackName: string;
ArtistName: string;
Album: string;
}
/**
* Every track in the library, in the order the backend lists them.
*
* The list arrives as a columnar `TrackTable` since #281 (repeated
* strings sent once, as indexes into `strings`), so it cannot be read
* as an array of tracks any more. This reads the three columns the
* specs use; `frontend/src/utils/track-table.ts` is the real decoder.
*/
export async function libraryTracks(page: Page): Promise<LibraryTrack[]> {
const t = await callBinding<{
strings: string[];
filePath: string[];
trackName: number[];
artistName: number[];
album: number[];
}>(page, 'library.Library.GetTrackTable', [0]);
return (t.filePath ?? []).map((FilePath, i) => ({
FilePath,
TrackName: t.strings[t.trackName[i]!] ?? '',
ArtistName: t.strings[t.artistName[i]!] ?? '',
Album: t.strings[t.album[i]!] ?? '',
}));
}
/** /**
* The binding calls the *app* made, newest last, as `pkg.Type.Method`. * The binding calls the *app* made, newest last, as `pkg.Type.Method`.
* *
@@ -18,6 +18,5 @@ export type {
ScanMetrics, ScanMetrics,
ScanWarning, ScanWarning,
Track, Track,
TrackMBIDs, TrackMBIDs
TrackTable
} from "./models.js"; } from "./models.js";
@@ -203,12 +203,16 @@ export function GetTrackMBIDs(filePath: string): $CancellablePromise<$models.Tra
} }
/** /**
* GetTrackTable returns every track in a library, or in all of them * GetTracks returns every track in a library, or in all of them when
* when libraryID is 0, as a TrackTable. An empty library is an empty * libraryID is 0.
* table, not an error. *
* The library id is a parameter rather than a second method because the
* two used to be separate queries, separate bindings and a branch at
* every call site - and the scoped form costs nothing (measured: 23 ms
* against 21 ms over 26k rows).
*/ */
export function GetTrackTable(libraryID: number): $CancellablePromise<$models.TrackTable> { export function GetTracks(libraryID: number): $CancellablePromise<$models.Track[] | null> {
return $Call.ByID(1179690926, libraryID); return $Call.ByID(933082923, libraryID);
} }
/** /**
@@ -218,21 +222,6 @@ export function GetTracksByGenre(genre: string, libraryID: number): $Cancellable
return $Call.ByID(1674220245, genre, libraryID); return $Call.ByID(1674220245, genre, libraryID);
} }
/**
* GetTracksByPaths returns whole tracks for the given file paths, in
* the order asked, dropping any path that is not in the library.
*
* It is how the frontend resolves the tracks a surface is actually
* showing (#279). Track details from the queue, a playlist or a smart
* playlist used to look the path up in the whole library's track
* array, which had to be loaded first — so it was fetched eagerly at
* startup, 20.5 MB at 26k tracks, to answer questions about a handful
* of rows.
*/
export function GetTracksByPaths(paths: string[] | null): $CancellablePromise<$models.Track[] | null> {
return $Call.ByID(3966945290, paths);
}
/** /**
* IsScanActive returns whether a scan is currently running. * IsScanActive returns whether a scan is currently running.
*/ */
@@ -239,60 +239,3 @@ export interface TrackMBIDs {
"releaseGroupMbid": string; "releaseGroupMbid": string;
"artistMbid": string; "artistMbid": string;
} }
/**
* TrackTable is every track in a library as the Tracks view uses it:
* one array per column, and every repeated string stored once (#281).
*
* GetTracks used to answer with one object per track, which at 26 138
* tracks was 20.5 MB of JSON — ~350 bytes a row of key names, four
* cover URLs identical across an album, and artist, album and genre
* strings repeated on every track of the album. Encoding it cost the
* backend ~170 MB of transient allocation and parsing it was the
* WebView's peak. Here the keys appear once, a repeated string is a
* small integer, and the columns the Tracks view does not read are not
* sent at all: LastPlayed and the three larger cover tiers belong to
* the details dialog, which fetches whole tracks by path.
*
* The projection is still trackFromRow's — each row goes through it —
* so this is an encoding of a Track, never a second description of
* one. frontend/src/utils/track-table.ts is the only decoder.
*/
export interface TrackTable {
/**
* Strings holds every distinct string value; a string column holds
* indexes into it. Index 0 is always "".
*/
"strings": string[] | null;
/**
* GenreSets holds every distinct genre list, as indexes into
* Strings; Genre holds an index into it per track.
*/
"genreSets": (number[] | null)[] | null;
"filePath": string[] | null;
"trackName": number[] | null;
"artistName": number[] | null;
"album": number[] | null;
"composer": number[] | null;
"fileType": number[] | null;
"genre": number[] | null;
"artistMbid": number[] | null;
"releaseGroupMbid": number[] | null;
"recordingMbid": number[] | null;
"coverArtSmall": number[] | null;
/**
* LengthMs is Track.TrackLength as the number it encodes.
*/
"lengthMs": number[] | null;
"trackNumber": number[] | null;
"discNumber": number[] | null;
"year": number[] | null;
"sampleRate": number[] | null;
"bitDepth": number[] | null;
"channels": number[] | null;
"bitrate": number[] | null;
"fileSize": number[] | null;
"playCount": number[] | null;
}
@@ -309,14 +309,6 @@ export function SearchLibrary(query: string): $CancellablePromise<$models.Candid
return $Call.ByID(3912116995, query); return $Call.ByID(3912116995, query);
} }
/**
* SuggestSmartPlaylistValues returns values of field present in the
* library that contain needle, for the rule editor's value box.
*/
export function SuggestSmartPlaylistValues(field: string, needle: string): $CancellablePromise<string[] | null> {
return $Call.ByID(3718198115, field, needle);
}
/** /**
* ToggleDefaultPlaylistTrack adds or removes a single track * ToggleDefaultPlaylistTrack adds or removes a single track
* from the default playlist. Returns true if the track is now * from the default playlist. Returns true if the track is now
+1 -9
View File
@@ -51,15 +51,7 @@
<div class="content-area"> <div class="content-area">
<main class="main-panel" data-active-view="tracks" data-testid="main-content" id="main-content" <main class="main-panel" data-active-view="tracks" data-testid="main-content" id="main-content"
tabindex="-1"> tabindex="-1">
<!-- The first-paint content of the main panel. It is <track-list></track-list>
`view-hidden` because whether Tracks is the view to
show is not known until `GetDefaultPage` answers, and a
seed that is visible is a seed that is *active* -- which
is what made the whole track list load at launch for a
landing on Home (#280). The first navigation decides:
it unhides and activates this element if the landing
view is Tracks, and otherwise replaces it. -->
<track-list class="view-hidden"></track-list>
</main> </main>
<queue-panel id="queue-panel"></queue-panel> <queue-panel id="queue-panel"></queue-panel>
</div> </div>
+2 -15
View File
@@ -200,18 +200,13 @@ const mainContent = document.getElementById('main-content');
// tracked as currentViewEl, is never hidden: two visible primary views // tracked as currentViewEl, is never hidden: two visible primary views
// splitting the main panel between them regardless of which is // splitting the main panel between them regardless of which is
// selected. // selected.
//
// It is deliberately not activated here. It is markup, not a decision:
// the shell does not yet know which view the launch lands on, and
// activating it starts the Tracks view's work — its data fetch, above
// all (#280) — for a launch that is about to land on Home. The first
// navigation is what activates whichever view it lands on.
if (mainContent) { if (mainContent) {
const initialTrackList = mainContent.querySelector('track-list'); const initialTrackList = mainContent.querySelector('track-list');
if (initialTrackList) { if (initialTrackList) {
viewCache.set('tracks', initialTrackList as HTMLElement); viewCache.set('tracks', initialTrackList as HTMLElement);
currentViewEl = initialTrackList as HTMLElement; currentViewEl = initialTrackList as HTMLElement;
activateView(currentViewEl);
} }
} }
@@ -585,14 +580,10 @@ async function handleNavigate(
} }
default: { default: {
const fallback = document.createElement('div'); const fallback = document.createElement('div');
const message = document.createElement('p');
fallback.style.padding = '1em'; fallback.style.padding = '1em';
fallback.style.color = 'var(--yj-text-secondary, #b3b3b3)'; fallback.style.color = 'var(--yj-text-secondary, #b3b3b3)';
// textContent, not innerHTML: `view` comes from a navigation fallback.innerHTML = `<p>Coming soon: ${view}</p>`;
// detail, which is app-supplied but not app-owned.
message.textContent = `Coming soon: ${view}`;
fallback.append(message);
mainContent.appendChild(fallback); mainContent.appendChild(fallback);
currentDetailEl = fallback; currentDetailEl = fallback;
} }
@@ -629,10 +620,6 @@ function warmViewChunks(): void {
/** requestIdleCallback where it exists; WebKit2GTK does not have it. */ /** requestIdleCallback where it exists; WebKit2GTK does not have it. */
function schedule(fn: () => void): void { function schedule(fn: () => void): void {
// SAFETY: `requestIdleCallback` is not in this project's DOM lib
// types, so it is read through an optional-property shape; the
// property is either absent (undefined) or the browser's own
// function, which is what the guard below checks.
const ric = ( const ric = (
window as unknown as { window as unknown as {
requestIdleCallback?: (cb: () => void) => number; requestIdleCallback?: (cb: () => void) => number;
@@ -7,11 +7,6 @@ import { designTokens } from '../../styles/tokens.css';
* keyboard navigation. Accepts a flat `options` string array, filters as * keyboard navigation. Accepts a flat `options` string array, filters as
* the user types, and emits `combobox-change` when a value is selected. * the user types, and emits `combobox-change` when a value is selected.
* *
* It also emits `combobox-input` with `{ text }` whenever the text it
* filters by changes (typing, and the reset to empty on focus), so a
* host whose options are too many to hand over at once can fetch the
* ones matching what was typed instead (#279).
*
* Key implementation detail: option `<li>` elements use `@mousedown` with * Key implementation detail: option `<li>` elements use `@mousedown` with
* `e.preventDefault()` so that the input's `blur` event does not close the * `e.preventDefault()` so that the input's `blur` event does not close the
* dropdown before the click registers. * dropdown before the click registers.
@@ -201,7 +196,6 @@ export class YjCombobox extends LitElement {
this.filterText = input.value; this.filterText = input.value;
this.open = true; this.open = true;
this.highlightedIndex = -1; this.highlightedIndex = -1;
this.announceInput();
} }
private handleFocus() { private handleFocus() {
@@ -209,17 +203,6 @@ export class YjCombobox extends LitElement {
this.filterText = ''; this.filterText = '';
this.open = true; this.open = true;
this.highlightedIndex = -1; this.highlightedIndex = -1;
this.announceInput();
}
private announceInput() {
this.dispatchEvent(
new CustomEvent('combobox-input', {
detail: { text: this.filterText },
bubbles: true,
composed: true,
}),
);
} }
private handleBlur() { private handleBlur() {
@@ -56,9 +56,12 @@ import {
createTrackCardDragImage, createTrackCardDragImage,
removeDragImage, removeDragImage,
} from '@utils/drag-image'; } from '@utils/drag-image';
import { libraryStore } from '@store/library-store';
import '@components/playlist-picker/playlist-picker.js'; import '@components/playlist-picker/playlist-picker.js';
import { showTrackDetailsForPath, showBatchTrackDetailsForPaths } from '@utils/track-details-opener.js'; import { loadTrackDetails } from '@utils/lazy-track-details.js';
import { tracksByFilePath, tracksForPaths } from '@utils/track-index.js';
import type { TrackDetails } from '@components/track-details/track-details.js'; import type { TrackDetails } from '@components/track-details/track-details.js';
import type { CoverArtUrls } from '@components/track-details/track-details.js';
import '@components/phantom-resolver/phantom-resolver.js'; import '@components/phantom-resolver/phantom-resolver.js';
import type { PhantomResolver } from '@components/phantom-resolver/phantom-resolver.js'; import type { PhantomResolver } from '@components/phantom-resolver/phantom-resolver.js';
import '@components/duplicate-tracks-dialog/duplicate-tracks-dialog.js'; import '@components/duplicate-tracks-dialog/duplicate-tracks-dialog.js';
@@ -755,21 +758,76 @@ export class PlaylistDetails
} }
private async openTrackDetails(filePath: string) { private async openTrackDetails(filePath: string) {
await showTrackDetailsForPath( const tracks = libraryStore.getCachedTracks();
() => this.trackDetailsDialog, const track = tracks
filePath, ? tracksByFilePath(tracks).get(filePath)
: undefined;
if (!track) return;
const ready = await loadTrackDetails(
() => void this.openTrackDetails(filePath), () => void this.openTrackDetails(filePath),
); );
if (!ready) return;
const coverArt = track.CoverArtPath
? {
coverArtPath: track.CoverArtPath,
coverArtSmall: track.CoverArtSmall,
coverArtMedium: track.CoverArtMedium,
coverArtLarge: track.CoverArtLarge,
}
: undefined;
this.trackDetailsDialog?.show(
track,
coverArt,
);
} }
private async openBatchTrackDetails( private async openBatchTrackDetails(
filePaths: string[], filePaths: string[],
) { ) {
await showBatchTrackDetailsForPaths( const cachedTracks =
() => this.trackDetailsDialog, libraryStore.getCachedTracks();
if (!cachedTracks) return;
const tracks = tracksForPaths(
cachedTracks,
filePaths, filePaths,
);
if (tracks.length === 0) return;
const ready = await loadTrackDetails(
() => void this.openBatchTrackDetails(filePaths), () => void this.openBatchTrackDetails(filePaths),
); );
if (!ready) return;
const first = tracks[0]!;
const albumNames = new Set(tracks.map((t) => t.Album));
let coverArt: CoverArtUrls | null = null;
let coverArtMixed = false;
if (albumNames.size === 1 && first.CoverArtPath) {
coverArt = {
coverArtPath: first.CoverArtPath,
coverArtSmall: first.CoverArtSmall,
coverArtMedium: first.CoverArtMedium,
coverArtLarge: first.CoverArtLarge,
};
} else if (albumNames.size > 1) {
coverArtMixed = true;
}
this.trackDetailsDialog?.showBatch(
tracks,
coverArt,
coverArtMixed,
);
} }
/** /**
@@ -51,8 +51,12 @@ import {
createTrackCardDragImage, createTrackCardDragImage,
removeDragImage, removeDragImage,
} from '@utils/drag-image'; } from '@utils/drag-image';
import { showTrackDetailsForPath, showBatchTrackDetailsForPaths } from '@utils/track-details-opener.js'; import { libraryStore } from '@store/library-store';
import type * as library from '@go/library/models.js';
import { loadTrackDetails } from '@utils/lazy-track-details.js';
import { tracksByFilePath } from '@utils/track-index.js';
import type { TrackDetails } from '@components/track-details/track-details.js'; import type { TrackDetails } from '@components/track-details/track-details.js';
import type { CoverArtUrls } from '@components/track-details/track-details.js';
import { import {
creditLink, creditLink,
trackLink, trackLink,
@@ -1538,33 +1542,89 @@ export class QueuePanel
} }
private async openTrackDetails(index: number) { private async openTrackDetails(index: number) {
const queueTrack = this.queue.tracks[index]; const queueTrack =
this.queue.tracks[index];
if (!queueTrack) return; if (!queueTrack) return;
await showTrackDetailsForPath( const tracks =
() => this.trackDetailsDialog, libraryStore.getCachedTracks();
const track = tracks
? tracksByFilePath(tracks).get(
queueTrack.filePath, queueTrack.filePath,
)
: undefined;
if (!track) return;
const ready = await loadTrackDetails(
() => void this.openTrackDetails(index), () => void this.openTrackDetails(index),
); );
if (!ready) return;
const coverArt = track.CoverArtPath
? {
coverArtPath: track.CoverArtPath,
coverArtSmall: track.CoverArtSmall,
coverArtMedium: track.CoverArtMedium,
coverArtLarge: track.CoverArtLarge,
}
: undefined;
this.trackDetailsDialog?.show(
track,
coverArt,
);
} }
private async openBatchTrackDetails( private async openBatchTrackDetails(
indices: number[], indices: number[],
) { ) {
const queueTracks = this.queue.tracks; const queueTracks = this.queue.tracks;
const filePaths: string[] = []; const cachedTracks =
libraryStore.getCachedTracks();
for (const i of indices) { if (!cachedTracks) return;
const path = queueTracks[i]?.filePath;
if (path != null) filePaths.push(path); const byPath = tracksByFilePath(cachedTracks);
const tracks = indices
.map((i) => queueTracks[i])
.filter((qt) => qt != null)
.map((qt) => byPath.get(qt.filePath))
.filter(
(t): t is library.Track =>
t != null,
);
if (tracks.length === 0) return;
const ready = await loadTrackDetails(
() => void this.openBatchTrackDetails(indices),
);
if (!ready) return;
const first = tracks[0]!;
const albumNames = new Set(tracks.map((t) => t.Album));
let coverArt: CoverArtUrls | null = null;
let coverArtMixed = false;
if (albumNames.size === 1 && first.CoverArtPath) {
coverArt = {
coverArtPath: first.CoverArtPath,
coverArtSmall: first.CoverArtSmall,
coverArtMedium: first.CoverArtMedium,
coverArtLarge: first.CoverArtLarge,
};
} else if (albumNames.size > 1) {
coverArtMixed = true;
} }
await showBatchTrackDetailsForPaths( this.trackDetailsDialog?.showBatch(
() => this.trackDetailsDialog, tracks,
filePaths, coverArt,
() => void this.openBatchTrackDetails(indices), coverArtMixed,
); );
} }
@@ -6,7 +6,6 @@ import { designTokens } from '../../styles/tokens.css';
import type { DragActiveDetail } from '@utils/drag-controller'; import type { DragActiveDetail } from '@utils/drag-controller';
import { ActiveViewController } from '@store/controllers/active-view-controller'; import { ActiveViewController } from '@store/controllers/active-view-controller';
import { ViewVisibilityController } from '@store/controllers/view-visibility-controller'; import { ViewVisibilityController } from '@store/controllers/view-visibility-controller';
import { libraryStore } from '@store/library-store';
import { VIEW_META } from '../../services/view-meta'; import { VIEW_META } from '../../services/view-meta';
import type { View } from '../../services/view-meta'; import type { View } from '../../services/view-meta';
@@ -353,10 +352,6 @@ export class AppSidebar extends LitElement {
: 'false'} : 'false'}
@click=${() => @click=${() =>
this.navigate(item.id)} this.navigate(item.id)}
@mouseenter=${() =>
this.prefetch(item.id)}
@focus=${() =>
this.prefetch(item.id)}
@dragover=${(e: DragEvent) => @dragover=${(e: DragEvent) =>
this.onNavDragOver( this.onNavDragOver(
e, e,
@@ -508,19 +503,6 @@ export class AppSidebar extends LitElement {
} }
} }
/**
* Start loading what this view draws, on hover or keyboard focus.
*
* The ~100 ms before the click is the whole point: #280 stopped
* fetching every collection at startup, and this is what keeps the
* view that *is* opened from paying the whole payload after the
* click. Nothing is awaited and nothing is reported here — the view
* itself reports a failure, and this is the same request.
*/
private prefetch(view: View) {
libraryStore.prefetch(view);
}
private navigate(view: View) { private navigate(view: View) {
// No optimistic highlight: the shell answers, and it answers // No optimistic highlight: the shell answers, and it answers
// synchronously in `handleNavigate` before it awaits anything. // synchronously in `handleNavigate` before it awaits anything.
@@ -52,8 +52,11 @@ import '@lit-labs/virtualizer';
import type { LitVirtualizer } from '@lit-labs/virtualizer'; import type { LitVirtualizer } from '@lit-labs/virtualizer';
import { flow } from '@lit-labs/virtualizer/layouts/flow.js'; import { flow } from '@lit-labs/virtualizer/layouts/flow.js';
import '@components/playlist-picker/playlist-picker.js'; import '@components/playlist-picker/playlist-picker.js';
import { showTrackDetailsForPath, showBatchTrackDetailsForPaths } from '@utils/track-details-opener.js'; import { loadTrackDetails } from '@utils/lazy-track-details.js';
import { tracksByFilePath, tracksForPaths } from '@utils/track-index.js';
import type { TrackDetails } from '@components/track-details/track-details.js'; import type { TrackDetails } from '@components/track-details/track-details.js';
import type { CoverArtUrls } from '@components/track-details/track-details.js';
import { libraryStore } from '@store/library-store';
import { formatMilliseconds } from '@utils/time'; import { formatMilliseconds } from '@utils/time';
import { import {
creditLink, creditLink,
@@ -1142,21 +1145,76 @@ export class SmartPlaylistDetails
} }
private async openTrackDetails(filePath: string) { private async openTrackDetails(filePath: string) {
await showTrackDetailsForPath( const tracks = libraryStore.getCachedTracks();
() => this.trackDetailsDialog, const track = tracks
filePath, ? tracksByFilePath(tracks).get(filePath)
: undefined;
if (!track) return;
const ready = await loadTrackDetails(
() => void this.openTrackDetails(filePath), () => void this.openTrackDetails(filePath),
); );
if (!ready) return;
const coverArt = track.CoverArtPath
? {
coverArtPath: track.CoverArtPath,
coverArtSmall: track.CoverArtSmall,
coverArtMedium: track.CoverArtMedium,
coverArtLarge: track.CoverArtLarge,
}
: undefined;
this.trackDetailsDialog?.show(
track,
coverArt,
);
} }
private async openBatchTrackDetails( private async openBatchTrackDetails(
filePaths: string[], filePaths: string[],
) { ) {
await showBatchTrackDetailsForPaths( const cachedTracks =
() => this.trackDetailsDialog, libraryStore.getCachedTracks();
if (!cachedTracks) return;
const tracks = tracksForPaths(
cachedTracks,
filePaths, filePaths,
);
if (tracks.length === 0) return;
const ready = await loadTrackDetails(
() => void this.openBatchTrackDetails(filePaths), () => void this.openBatchTrackDetails(filePaths),
); );
if (!ready) return;
const first = tracks[0]!;
const albumNames = new Set(tracks.map((t) => t.Album));
let coverArt: CoverArtUrls | null = null;
let coverArtMixed = false;
if (albumNames.size === 1 && first.CoverArtPath) {
coverArt = {
coverArtPath: first.CoverArtPath,
coverArtSmall: first.CoverArtSmall,
coverArtMedium: first.CoverArtMedium,
coverArtLarge: first.CoverArtLarge,
};
} else if (albumNames.size > 1) {
coverArtMixed = true;
}
this.trackDetailsDialog?.showBatch(
tracks,
coverArt,
coverArtMixed,
);
} }
// ================================================================= // =================================================================
@@ -1,12 +1,8 @@
import { LitElement, html, css, nothing } from 'lit'; import { LitElement, html, css, nothing } from 'lit';
import { customElement, property, state } from 'lit/decorators.js'; import { customElement, property, state } from 'lit/decorators.js';
import * as library from '@go/library/models.js'; import * as library from '@go/library/models.js';
import { import { PreviewSmartPlaylist } from '@go/playlist/service.js';
PreviewSmartPlaylist, import { libraryStore } from '@store/library-store';
SuggestSmartPlaylistValues,
} from '@go/playlist/service.js';
import { list } from '@utils/binding';
import { LRUMap } from '@utils/lru-map';
import { describeError } from '@utils/describe-error'; import { describeError } from '@utils/describe-error';
import { designTokens } from '../../styles/tokens.css'; import { designTokens } from '../../styles/tokens.css';
import '@components/combobox/combobox.ts'; import '@components/combobox/combobox.ts';
@@ -95,23 +91,51 @@ function formatOperatorLabel(op: string): string {
return op.replace(/_/g, ' '); return op.replace(/_/g, ' ');
} }
/** /** Returns autocomplete suggestions for a given field from libraryStore. */
* Fields whose value box suggests values present in the library. Must function getAutocompleteOptions(field: string): string[] {
* match `suggestFields` in backend/smartplaylist; any other field gets switch (field) {
* a plain box. case 'artist':
* return libraryStore.getCachedArtists()?.map((a) => a.Name) ?? [];
* Suggestions are asked for as the user types (#279): they used to be case 'genre':
* built from libraryStore's whole-library arrays, which made this return libraryStore.getCachedGenres()?.map((g) => g.Name) ?? [];
* editor one more reason to load every track at startup, and gave it case 'album':
* empty lists whenever those arrays had not landed. return libraryStore.getCachedAlbums()?.map((a) => a.Name) ?? [];
*/ case 'title': {
const SUGGEST_FIELDS = new Set([ const tracks = libraryStore.getCachedTracks();
'title', 'artist', 'album', 'genre', 'composer', 'file_type', if (!tracks) return [];
'year', 'release_year', return [...new Set(tracks.map((t) => t.TrackName).filter(Boolean))];
]); }
case 'composer': {
/** How long typing must pause before suggestions are asked for. */ const tracks = libraryStore.getCachedTracks();
const SUGGEST_DEBOUNCE_MS = 120; if (!tracks) return [];
return [...new Set(tracks.map((t) => t.Composer).filter(Boolean))];
}
case 'file_type': {
const tracks = libraryStore.getCachedTracks();
if (!tracks) return [];
return [...new Set(tracks.map((t) => t.FileType).filter(Boolean))];
}
case 'year':
case 'release_year': {
// Both year fields draw suggestions from the set of years
// present in the library. The cached Track only carries the
// display (original) year, so it seeds both datalists — the
// list is just a hint, and the real filter runs server-side.
const tracks = libraryStore.getCachedTracks();
if (!tracks) return [];
return [
...new Set(
tracks
.map((t) => t.Year)
.filter((y) => y > 0)
.map(String),
),
].sort();
}
default:
return [];
}
}
/** /**
* Overrides for fields whose title-cased name would be ambiguous. The * Overrides for fields whose title-cased name would be ambiguous. The
@@ -182,17 +206,6 @@ export class SmartPlaylistEditor extends LitElement {
private previewTimer: ReturnType<typeof setTimeout> | null = null; private previewTimer: ReturnType<typeof setTimeout> | null = null;
/** The value suggestions each rule row is showing, by row index. */
@state() private suggestions = new Map<number, readonly string[]>();
private suggestTimer: ReturnType<typeof setTimeout> | null = null;
/** Answers already fetched this session, by field and typed text. */
private suggestCache = new LRUMap<string, readonly string[]>(100);
/** The latest request per row, so a slow answer cannot overwrite a newer one. */
private suggestWanted = new Map<number, string>();
// ── Styles ────────────────────────────────────────────────────── // ── Styles ──────────────────────────────────────────────────────
static override styles = [ static override styles = [
@@ -586,7 +599,6 @@ export class SmartPlaylistEditor extends LitElement {
// Reset value when field changes to avoid stale autocomplete data // Reset value when field changes to avoid stale autocomplete data
row.value = ''; row.value = '';
row.value2 = ''; row.value2 = '';
this.dropSuggestions();
this.ruleRows = [...this.ruleRows]; this.ruleRows = [...this.ruleRows];
this.onRulesChanged(); this.onRulesChanged();
@@ -616,56 +628,6 @@ export class SmartPlaylistEditor extends LitElement {
this.onRulesChanged(); this.onRulesChanged();
} }
/** Ask for the values of row `index`'s field that contain `text`. */
private requestSuggestions(index: number, text: string) {
const field = this.ruleRows[index]?.field ?? '';
if (!SUGGEST_FIELDS.has(field)) return;
const key = `${field}\u0000${text}`;
this.suggestWanted.set(index, key);
const cached = this.suggestCache.get(key);
if (cached) {
this.setSuggestions(index, cached);
return;
}
if (this.suggestTimer !== null) clearTimeout(this.suggestTimer);
this.suggestTimer = setTimeout(() => {
this.suggestTimer = null;
void list(SuggestSmartPlaylistValues(field, text))
.then((values) => {
this.suggestCache.set(key, values);
if (this.suggestWanted.get(index) === key) {
this.setSuggestions(index, values);
}
})
.catch((err: unknown) => {
// A suggestion is a hint; the box still takes any text.
console.error('smart playlist: suggestions failed', err);
});
}, SUGGEST_DEBOUNCE_MS);
}
private setSuggestions(index: number, values: readonly string[]) {
const next = new Map(this.suggestions);
next.set(index, values);
this.suggestions = next;
}
/** Row indexes and fields changed: what was suggested no longer applies. */
private dropSuggestions() {
this.suggestions = new Map();
this.suggestWanted.clear();
}
private updateValue2(index: number, newValue: string) { private updateValue2(index: number, newValue: string) {
const row = this.ruleRows[index]; const row = this.ruleRows[index];
if (!row) return; if (!row) return;
@@ -682,7 +644,6 @@ export class SmartPlaylistEditor extends LitElement {
private removeRule(index: number) { private removeRule(index: number) {
if (this.ruleRows.length <= 1) return; if (this.ruleRows.length <= 1) return;
this.ruleRows = this.ruleRows.filter((_, i) => i !== index); this.ruleRows = this.ruleRows.filter((_, i) => i !== index);
this.dropSuggestions();
this.onRulesChanged(); this.onRulesChanged();
} }
@@ -931,9 +892,7 @@ export class SmartPlaylistEditor extends LitElement {
` `
: html` : html`
<yj-combobox <yj-combobox
.options=${this.suggestions.get(index) ?? []} .options=${getAutocompleteOptions(row.field)}
@combobox-input=${(e: CustomEvent<{ text: string }>) =>
this.requestSuggestions(index, e.detail.text)}
.value=${row.value} .value=${row.value}
placeholder=${isAnyOf placeholder=${isAnyOf
? 'Comma-separated values' ? 'Comma-separated values'
@@ -22,9 +22,7 @@ import {
import { GetTrackMBIDs } from '@go/library/library.js'; import { GetTrackMBIDs } from '@go/library/library.js';
type TrackMBIDs = library.TrackMBIDs; type TrackMBIDs = library.TrackMBIDs;
import { ImageFilePicker, ReadFile } from '@go/frontendutil/frontendutil.js'; import { ImageFilePicker, ReadFile } from '@go/frontendutil/frontendutil.js';
import { trackCache } from '../../store/track-cache'; import { libraryStore } from '../../store/library-store';
import { batchCoverArt } from '@utils/track-details-opener.js';
import type { ListTrack } from '@utils/track-table';
import { EventsOn } from '@runtime/runtime'; import { EventsOn } from '@runtime/runtime';
import { Events } from '../../events'; import { Events } from '../../events';
@@ -89,11 +87,7 @@ export class TrackDetails extends LitElement {
// -- Batch-specific state -- // -- Batch-specific state --
@state() private batchMode = false; @state() private batchMode = false;
// `ListTrack`, not `Track`: every field the merged-values view reads @state() private batchTracks: library.Track[] = [];
// is on the list's rows, so a caller that is already showing them
// can open this dialog without fetching anything (#281). The
// openers that hold no rows ask `trackCache` instead.
@state() private batchTracks: ListTrack[] = [];
@state() private batchFilePaths: string[] = []; @state() private batchFilePaths: string[] = [];
@state() private batchCoverArtMixed = false; @state() private batchCoverArtMixed = false;
@state() private batchProgress: { @state() private batchProgress: {
@@ -141,12 +135,12 @@ export class TrackDetails extends LitElement {
/** Open the dialog for batch editing multiple tracks. */ /** Open the dialog for batch editing multiple tracks. */
showBatch( showBatch(
tracks: readonly ListTrack[], tracks: library.Track[],
coverArt: CoverArtUrls | null, coverArt: CoverArtUrls | null,
coverArtMixed: boolean, coverArtMixed: boolean,
): void { ): void {
this.batchMode = true; this.batchMode = true;
this.batchTracks = [...tracks]; this.batchTracks = tracks;
this.batchFilePaths = tracks.map( this.batchFilePaths = tracks.map(
(t) => t.FilePath, (t) => t.FilePath,
); );
@@ -1323,13 +1317,10 @@ export class TrackDetails extends LitElement {
const container = img.parentElement; const container = img.parentElement;
if (container) { if (container) {
const placeholder = document.createElement('div'); container.innerHTML =
const icon = document.createElement('wa-icon'); '<div class="cover-placeholder">' +
'<wa-icon name="music"></wa-icon>' +
placeholder.className = 'cover-placeholder'; '</div>';
icon.setAttribute('name', 'music');
placeholder.append(icon);
container.replaceChildren(placeholder);
} }
}; };
@@ -1699,11 +1690,20 @@ export class TrackDetails extends LitElement {
// invalidation, which refreshes all other views. // invalidation, which refreshes all other views.
this.exitEditMode(); this.exitEditMode();
// Re-read this one track so the dialog shows the // Re-fetch track and album data so the dialog
// values and cover art just written. Not the whole // shows updated values and cover art. The store
// library (#279): the views that hold it refetch on // invalidation is already in-flight from the event;
// the event, and only if something is showing them. // these calls await the pending fetch or start one.
const [updated] = await trackCache.refresh([filePath]); // Awaited for the side effect of refreshing the
// store; the album list is consumed elsewhere.
const [tracks] = await Promise.all([
libraryStore.getTracks(),
libraryStore.getAlbums(),
]);
const updated = tracks.find(
(t) => t.FilePath === filePath,
);
if (updated) { if (updated) {
this.track = updated; this.track = updated;
@@ -1828,18 +1828,39 @@ export class TrackDetails extends LitElement {
this.errorMessage = ''; this.errorMessage = '';
this.cleanupPendingCoverArt(); this.cleanupPendingCoverArt();
// Re-read the tracks just written, in the order the batch // Refresh data from library store; album list is
// holds them (#279) — not the whole library to filter. // refreshed for side effects only.
// `refresh`, because the write's event may not have reached const [tracks] = await Promise.all([
// the cache before its own result did. libraryStore.getTracks(),
const refreshed = await trackCache.refresh(this.batchFilePaths); libraryStore.getAlbums(),
]);
// Re-resolve batch tracks.
const pathSet = new Set(this.batchFilePaths);
const refreshed = tracks.filter((t) =>
pathSet.has(t.FilePath),
);
this.batchTracks = refreshed; this.batchTracks = refreshed;
const { coverArt, mixed } = batchCoverArt(refreshed); // Re-resolve cover art state.
const first = refreshed[0];
const albumNames = new Set(refreshed.map((t) => t.Album));
this.coverArt = coverArt; if (albumNames.size === 1 && first?.CoverArtPath) {
this.batchCoverArtMixed = mixed; this.coverArt = {
coverArtPath: first.CoverArtPath,
coverArtSmall: first.CoverArtSmall,
coverArtMedium: first.CoverArtMedium,
coverArtLarge: first.CoverArtLarge,
};
this.batchCoverArtMixed = false;
} else if (albumNames.size > 1) {
this.coverArt = null;
this.batchCoverArtMixed = true;
} else {
this.coverArt = null;
this.batchCoverArtMixed = false;
}
}; };
// -- Shared edit logic -- // -- Shared edit logic --
@@ -2082,8 +2103,6 @@ export class TrackDetails extends LitElement {
// as a base64-encoded string (standard encoding/json // as a base64-encoded string (standard encoding/json
// behaviour for []byte). // behaviour for []byte).
const result = await ReadFile(filePath); const result = await ReadFile(filePath);
// SAFETY: the binding is typed as the Go []byte, but the
// wire value is the base64 string encoding/json made of it.
const b64 = result as unknown as string; const b64 = result as unknown as string;
const binary = atob(b64); const binary = atob(b64);
const bytes = new Uint8Array(binary.length); const bytes = new Uint8Array(binary.length);
@@ -2138,7 +2157,7 @@ export class TrackDetails extends LitElement {
key: string; key: string;
label: string; label: string;
type: 'text' | 'number'; type: 'text' | 'number';
extract: (t: ListTrack) => string; extract: (t: library.Track) => string;
}> = [ }> = [
{ {
key: 'title', key: 'title',
@@ -2222,7 +2241,7 @@ export class TrackDetails extends LitElement {
private countDistinctValues(key: string): number { private countDistinctValues(key: string): number {
const extractMap: Record< const extractMap: Record<
string, string,
(t: ListTrack) => string (t: library.Track) => string
> = { > = {
title: (t) => t.TrackName ?? '', title: (t) => t.TrackName ?? '',
artist: (t) => t.ArtistName ?? '', artist: (t) => t.ArtistName ?? '',
@@ -2246,7 +2265,7 @@ export class TrackDetails extends LitElement {
if (!extract) return 0; if (!extract) return 0;
const unique = new Set( const unique = new Set(
this.batchTracks.map((t) => extract(t)), this.batchTracks.map(extract),
); );
return unique.size; return unique.size;
+10 -10
View File
@@ -1,4 +1,4 @@
import type { ListTrack } from '@utils/track-table'; import type * as library from '@go/library/models.js';
import { import {
formatSampleRate, formatSampleRate,
formatBitDepth, formatBitDepth,
@@ -45,7 +45,7 @@ export interface ColumnDef {
*/ */
configurable?: boolean; configurable?: boolean;
/** Extracts the display value from a track. */ /** Extracts the display value from a track. */
accessor: (track: ListTrack) => string; accessor: (track: library.Track) => string;
/** Default CSS width (used when no saved width exists). */ /** Default CSS width (used when no saved width exists). */
defaultWidth: string; defaultWidth: string;
/** Text alignment. Defaults to left. */ /** Text alignment. Defaults to left. */
@@ -60,15 +60,15 @@ export interface ColumnDef {
* needs it, which is why it is optional rather than a second * needs it, which is why it is optional rather than a second
* required parameter on all of them. * required parameter on all of them.
*/ */
renderCell?: (track: ListTrack, term?: string) => unknown; renderCell?: (track: library.Track, term?: string) => unknown;
/** /**
* Comparison function for sorting two tracks by this column. * Comparison function for sorting two tracks by this column.
* Returns negative if a < b, positive if a > b, zero if equal. * Returns negative if a < b, positive if a > b, zero if equal.
* If omitted the column is not sortable. * If omitted the column is not sortable.
*/ */
comparator?: ( comparator?: (
a: ListTrack, a: library.Track,
b: ListTrack, b: library.Track,
) => number; ) => number;
} }
@@ -79,7 +79,7 @@ export const COLUMN_DEFS: Record<string, ColumnDef> = {
label: 'Art', label: 'Art',
accessor: () => '', accessor: () => '',
defaultWidth: '36px', defaultWidth: '36px',
renderCell: (track: ListTrack) => { renderCell: (track: library.Track) => {
// `perf.M3`. This rendered `CoverArtPath` — the *original* // `perf.M3`. This rendered `CoverArtPath` — the *original*
// embedded artwork, commonly 1500×1500 and several hundred // embedded artwork, commonly 1500×1500 and several hundred
// kB — scaled by CSS into a 24 px box, while the 100 px // kB — scaled by CSS into a 24 px box, while the 100 px
@@ -90,10 +90,10 @@ export const COLUMN_DEFS: Record<string, ColumnDef> = {
// `cover-grid.getCoverUrl()` has picked the right tier all // `cover-grid.getCoverUrl()` has picked the right tier all
// along; this is the same rule for a much smaller box, with // along; this is the same rule for a much smaller box, with
// the two attributes that keep the decode off the scroll // the two attributes that keep the decode off the scroll
// path. The list carries only this tier (#281): every tier // path.
// is derived from the same file, so it is set whenever any const src = track.CoverArtSmall
// of them would be. || track.CoverArtMedium
const src = track.CoverArtSmall; || track.CoverArtPath;
if (!src) return nothing; if (!src) return nothing;
@@ -1,4 +1,4 @@
import type { ListTrack } from '@utils/track-table'; import type * as library from '@go/library/models.js';
import { html } from 'lit'; import { html } from 'lit';
import type { TemplateResult } from 'lit'; import type { TemplateResult } from 'lit';
@@ -99,7 +99,7 @@ function matchQuality(
* fields are always included on top of these. * fields are always included on top of these.
*/ */
function scoreTrack( function scoreTrack(
track: ListTrack, track: library.Track,
termLower: string, termLower: string,
columns: ColumnDef[], columns: ColumnDef[],
): number { ): number {
@@ -147,7 +147,7 @@ function scoreTrack(
/** A track paired with its relevance score. */ /** A track paired with its relevance score. */
export interface RankedTrack { export interface RankedTrack {
track: ListTrack; track: library.Track;
score: number; score: number;
} }
@@ -166,10 +166,10 @@ export interface RankedTrack {
* `scores` (Map of FilePath → relevance score). * `scores` (Map of FilePath → relevance score).
*/ */
export function rankTracks( export function rankTracks(
tracks: ListTrack[], tracks: library.Track[],
term: string, term: string,
activeColumns: ColumnDef[], activeColumns: ColumnDef[],
): { tracks: ListTrack[]; scores: Map<string, number> } { ): { tracks: library.Track[]; scores: Map<string, number> } {
const termLower = term.toLowerCase(); const termLower = term.toLowerCase();
const ranked: RankedTrack[] = []; const ranked: RankedTrack[] = [];
@@ -188,7 +188,7 @@ export function rankTracks(
// Sort descending by score (highest relevance first). // Sort descending by score (highest relevance first).
ranked.sort((a, b) => b.score - a.score); ranked.sort((a, b) => b.score - a.score);
const result: ListTrack[] = []; const result: library.Track[] = [];
const scores = new Map<string, number>(); const scores = new Map<string, number>();
for (const r of ranked) { for (const r of ranked) {
@@ -1,5 +1,5 @@
import type { ListTrack } from '@utils/track-table'; import * as library from '@go/library/models.js';
import { LitElement, html, svg, css, nothing, type TemplateResult } from 'lit'; import { LitElement, html, svg, css, nothing } from 'lit';
import { designTokens } from '../../styles/tokens.css'; import { designTokens } from '../../styles/tokens.css';
import { srOnly } from '../../styles/sr-only.css'; import { srOnly } from '../../styles/sr-only.css';
import { import {
@@ -80,10 +80,11 @@ import { describeError } from '@utils/describe-error';
import { notificationStore } from '@store/notification-store'; import { notificationStore } from '@store/notification-store';
import { confirmAction } from '@components/confirm-dialog/confirm-dialog'; import { confirmAction } from '@components/confirm-dialog/confirm-dialog';
import { RemoveFromLibrary } from '@go/library/library.js'; import { RemoveFromLibrary } from '@go/library/library.js';
import { showTrackDetailsForPath, showBatchTrackDetails } from '@utils/track-details-opener.js'; import { loadTrackDetails } from '@utils/lazy-track-details.js';
import { tracksByFilePath, tracksForPaths } from '@utils/track-index.js'; import { tracksByFilePath, tracksForPaths } from '@utils/track-index.js';
import '@components/playlist-picker/playlist-picker.js'; import '@components/playlist-picker/playlist-picker.js';
import type { TrackDetails } from '@components/track-details/track-details.js'; import type { TrackDetails } from '@components/track-details/track-details.js';
import type { CoverArtUrls } from '@components/track-details/track-details.js';
import { import {
ICON_PLAY, ICON_PLAY,
ICON_PLAYLIST, ICON_PLAYLIST,
@@ -154,7 +155,7 @@ export class TrackList
* parent is responsible for reloading when data changes. * parent is responsible for reloading when data changes.
*/ */
@property({ type: Array, attribute: false }) @property({ type: Array, attribute: false })
externalTracks?: ListTrack[]; externalTracks?: library.Track[];
/** /**
* What a host embedding this list (e.g. `genre-details`) should say * What a host embedding this list (e.g. `genre-details`) should say
@@ -198,7 +199,7 @@ export class TrackList
private lastSearchTerm = ''; private lastSearchTerm = '';
/** Tracks the store's cached array reference to detect refreshes. */ /** Tracks the store's cached array reference to detect refreshes. */
private lastTracksRef: ListTrack[] | null = private lastTracksRef: library.Track[] | null =
null; null;
/** /**
@@ -241,7 +242,7 @@ export class TrackList
} }
@state() @state()
private tracks: ListTrack[] = []; private tracks: library.Track[] = [];
@query('#context-menu') @query('#context-menu')
private contextMenuPopup!: MenuSurface; private contextMenuPopup!: MenuSurface;
@@ -268,16 +269,16 @@ export class TrackList
private lastActiveTrackPath: string | null = null; private lastActiveTrackPath: string | null = null;
// -- Memoisation caches for filtered / sorted tracks -- // -- Memoisation caches for filtered / sorted tracks --
private cachedFilteredTracks: ListTrack[] = []; private cachedFilteredTracks: library.Track[] = [];
private cachedSortedTracks: ListTrack[] = []; private cachedSortedTracks: library.Track[] = [];
private cachedRelevanceScores = new Map< private cachedRelevanceScores = new Map<
string, string,
number number
>(); >();
private prevFilterTracks: ListTrack[] = []; private prevFilterTracks: library.Track[] = [];
private prevFilterTerm = ''; private prevFilterTerm = '';
private prevFilterColIds = ''; private prevFilterColIds = '';
private prevSortFiltered: ListTrack[] = []; private prevSortFiltered: library.Track[] = [];
private prevSortField: string | null = null; private prevSortField: string | null = null;
private prevSortDir: SortDirection = 'asc'; private prevSortDir: SortDirection = 'asc';
@@ -522,7 +523,7 @@ export class TrackList
} }
} }
private computeFilteredTracks(): ListTrack[] { private computeFilteredTracks(): library.Track[] {
const term = this.searchCtrl.term; const term = this.searchCtrl.term;
if (!term) { if (!term) {
@@ -542,7 +543,7 @@ export class TrackList
return result.tracks; return result.tracks;
} }
private computeSortedTracks(): ListTrack[] { private computeSortedTracks(): library.Track[] {
const tracks = this.cachedFilteredTracks; const tracks = this.cachedFilteredTracks;
const hasSearch = const hasSearch =
this.cachedRelevanceScores.size > 0; this.cachedRelevanceScores.size > 0;
@@ -1336,6 +1337,11 @@ export class TrackList
super.connectedCallback(); super.connectedCallback();
this.restoreSortPreferences(); this.restoreSortPreferences();
if (this.externalTracks) {
this.tracks = this.externalTracks;
} else {
this.loadTracks();
}
this.resizeObserver = new ResizeObserver( this.resizeObserver = new ResizeObserver(
() => { () => {
this.onHostResize(); this.onHostResize();
@@ -1385,33 +1391,10 @@ export class TrackList
this.resizeObserver = null; this.resizeObserver = null;
} }
/**
* Fetch the list when this becomes the view on screen (#280).
*
* Not on connection: `index.html` renders a `<track-list>` as the
* main panel's first-paint content, so a connection-time fetch was
* the whole library loaded at launch for a landing on Home — 12 MB
* at 50 000 tracks, and the backend's peak RSS with it. The shell
* activates this element only when a navigation lands on Tracks.
*
* Called on *every* activation, not just the first: the store may
* have refetched while this view was off screen, and `getTracks()`
* answers from its cache when nothing changed.
*/
protected override onViewActivate(): void {
if (this.externalTracks) {
this.tracks = this.externalTracks;
} else {
void this.loadTracks();
}
this.attachListListeners();
}
/** Document-level listeners belong to the *visible* list. A cached /** Document-level listeners belong to the *visible* list. A cached
* list is never disconnected, so this is the only place they can be * list is never disconnected, so this is the only place they can be
* taken down again. */ * taken down again. */
private attachListListeners(): void { protected override onViewActivate(): void {
this.listenWhileActive(document, 'click', this.clearSelectionHandler); this.listenWhileActive(document, 'click', this.clearSelectionHandler);
this.listenWhileActive( this.listenWhileActive(
document, document,
@@ -1578,10 +1561,9 @@ export class TrackList
this.selection.clear(); this.selection.clear();
} }
// Re-fetch when the store delivers fresh data after an // Re-fetch when the store delivers fresh
// invalidation. Off screen the list does nothing with it, and // data after eager refetch on invalidation.
// the next activation re-reads the store (#280). if (!this.externalTracks) {
if (!this.externalTracks && this.viewActive) {
const cached = const cached =
this.libraryCtrl.cachedTracks; this.libraryCtrl.cachedTracks;
@@ -1752,7 +1734,7 @@ export class TrackList
*/ */
private resolveTrackFromEvent( private resolveTrackFromEvent(
e: Event, e: Event,
): { track: ListTrack; index: number } | null { ): { track: library.Track; index: number } | null {
const row = (e.target as HTMLElement).closest( const row = (e.target as HTMLElement).closest(
'.track-row', '.track-row',
) as HTMLElement | null; ) as HTMLElement | null;
@@ -1893,7 +1875,7 @@ export class TrackList
private onTrackRowClick( private onTrackRowClick(
e: MouseEvent, e: MouseEvent,
track: ListTrack, track: library.Track,
index: number, index: number,
) { ) {
// Clicking is also how the keyboard's starting point is chosen: // Clicking is also how the keyboard's starting point is chosen:
@@ -1902,7 +1884,7 @@ export class TrackList
this.selection.handleItemClick(e, track.FilePath, index); this.selection.handleItemClick(e, track.FilePath, index);
} }
private onTrackRowDblClick(_track: ListTrack, index: number) { private onTrackRowDblClick(_track: library.Track, index: number) {
this.selection.clear(); this.selection.clear();
this.playFromRow(index); this.playFromRow(index);
} }
@@ -1943,7 +1925,7 @@ export class TrackList
); );
} }
private onTrackContextMenu(e: MouseEvent, track: ListTrack) { private onTrackContextMenu(e: MouseEvent, track: library.Track) {
e.preventDefault(); e.preventDefault();
e.stopPropagation(); e.stopPropagation();
@@ -1957,7 +1939,7 @@ export class TrackList
private onTrackDragStart = ( private onTrackDragStart = (
e: DragEvent, e: DragEvent,
track: ListTrack, track: library.Track,
) => { ) => {
// Gather file paths: all selected if this track is selected, // Gather file paths: all selected if this track is selected,
// otherwise just the dragged track. // otherwise just the dragged track.
@@ -2147,26 +2129,74 @@ export class TrackList
this.ctxMenu.close(); this.ctxMenu.close();
} }
// The list's rows do not carry what the dialog shows (#281), so
// the dialog gets whole tracks by path, like every other opener.
private async openTrackDetails(filePath: string) { private async openTrackDetails(filePath: string) {
await showTrackDetailsForPath( const track = tracksByFilePath(this.tracks).get(
() => this.trackDetailsDialog,
filePath, filePath,
);
if (!track) return;
const ready = await loadTrackDetails(
() => void this.openTrackDetails(filePath), () => void this.openTrackDetails(filePath),
); );
if (!ready) return;
const coverArt = track.CoverArtPath
? {
coverArtPath: track.CoverArtPath,
coverArtSmall: track.CoverArtSmall,
coverArtMedium: track.CoverArtMedium,
coverArtLarge: track.CoverArtLarge,
}
: undefined;
this.trackDetailsDialog?.show(
track,
coverArt,
);
} }
private async openBatchTrackDetails( private async openBatchTrackDetails(
filePaths: string[], filePaths: string[],
) { ) {
// The rows in hand are what the dialog reads, so a "select all" const tracks = tracksForPaths(
// on 50 000 tracks opens it without asking the backend for them. this.tracks,
await showBatchTrackDetails( filePaths,
() => this.trackDetailsDialog, );
tracksForPaths(this.tracks, filePaths),
if (tracks.length === 0) return;
const ready = await loadTrackDetails(
() => void this.openBatchTrackDetails(filePaths), () => void this.openBatchTrackDetails(filePaths),
); );
if (!ready) return;
// Use cover art from the first track. If all tracks share
// the same album, they share the same art.
const first = tracks[0]!;
let coverArt: CoverArtUrls | null = null;
let coverArtMixed = false;
const albumNames = new Set(tracks.map((t) => t.Album));
if (albumNames.size === 1 && first.CoverArtPath) {
coverArt = {
coverArtPath: first.CoverArtPath,
coverArtSmall: first.CoverArtSmall,
coverArtMedium: first.CoverArtMedium,
coverArtLarge: first.CoverArtLarge,
};
} else if (albumNames.size > 1) {
coverArtMixed = true;
}
this.trackDetailsDialog?.showBatch(
tracks,
coverArt,
coverArtMixed,
);
} }
// ================================================================= // =================================================================
@@ -2244,7 +2274,7 @@ export class TrackList
this.saveSortPreferences(); this.saveSortPreferences();
} }
private isActiveTrack(track: ListTrack): boolean { private isActiveTrack(track: library.Track): boolean {
const currentTrack = this.player.currentTrack; const currentTrack = this.player.currentTrack;
if (!currentTrack) return false; if (!currentTrack) return false;
@@ -2253,9 +2283,9 @@ export class TrackList
} }
private renderTrackRow = ( private renderTrackRow = (
track: ListTrack, track: library.Track,
index: number, index: number,
): TemplateResult => { ): unknown => {
const active = this.isActiveTrack(track); const active = this.isActiveTrack(track);
const selected = this.selection.isSelected( const selected = this.selection.isSelected(
track.FilePath, track.FilePath,
@@ -2538,7 +2568,7 @@ export class TrackList
scroller scroller
.items=${visibleTracks} .items=${visibleTracks}
.renderItem=${this.renderTrackRow} .renderItem=${this.renderTrackRow}
.keyFunction=${(track: ListTrack) => track.FilePath} .keyFunction=${(track: library.Track) => track.FilePath}
.layout=${this.rowLayout} .layout=${this.rowLayout}
></lit-virtualizer> ></lit-virtualizer>
`} `}
@@ -1,7 +1,6 @@
import type { ReactiveController, ReactiveControllerHost } from 'lit'; import type { ReactiveController, ReactiveControllerHost } from 'lit';
import type * as library from '@go/library/models.js'; import type * as library from '@go/library/models.js';
import { libraryStore } from '../library-store'; import { libraryStore } from '../library-store';
import type { ListTrack } from '@utils/track-table';
type ViewName = 'tracks' | 'albums' | 'artists' | 'genres'; type ViewName = 'tracks' | 'albums' | 'artists' | 'genres';
@@ -56,7 +55,7 @@ export class LibraryController implements ReactiveController {
// DATA ACCESS // DATA ACCESS
// =================================================================== // ===================================================================
async getTracks(): Promise<ListTrack[]> { async getTracks(): Promise<library.Track[]> {
return libraryStore.getTracks(); return libraryStore.getTracks();
} }
@@ -86,7 +85,7 @@ export class LibraryController implements ReactiveController {
); );
} }
get cachedTracks(): ListTrack[] | null { get cachedTracks(): library.Track[] | null {
return libraryStore.getCachedTracks(); return libraryStore.getCachedTracks();
} }
+64 -244
View File
@@ -1,6 +1,6 @@
import { EventsOn } from '@runtime/runtime'; import { EventsOn } from '@runtime/runtime';
import { import {
GetTrackTable, GetTracks,
GetAlbums, GetAlbums,
GetArtists, GetArtists,
GetGenres, GetGenres,
@@ -9,8 +9,6 @@ import {
} from '@go/library/library.js'; } from '@go/library/library.js';
import type * as library from '@go/library/models.js'; import type * as library from '@go/library/models.js';
import { list } from '@utils/binding'; import { list } from '@utils/binding';
import { trackCache } from './track-cache';
import { decodeTrackTable, type ListTrack } from '@utils/track-table';
import { Events } from '../events'; import { Events } from '../events';
type ViewName = 'tracks' | 'albums' | 'artists' | 'genres'; type ViewName = 'tracks' | 'albums' | 'artists' | 'genres';
@@ -29,16 +27,8 @@ const COVER_SIZE_DEFAULT = 176;
/** localStorage key for persisted cover size. */ /** localStorage key for persisted cover size. */
const COVER_SIZE_KEY = 'cover-grid-size'; const COVER_SIZE_KEY = 'cover-grid-size';
/**
* How long the small-collection warm-up will wait for idle before
* running anyway. It is speculative, but a busy main thread must not
* mean Albums is slow to open — the timeout is the promise that it is
* only ever deferred, never skipped.
*/
const WARM_IDLE_TIMEOUT_MS = 3_000;
class LibraryStore { class LibraryStore {
private tracks: ListTrack[] | null = null; private tracks: library.Track[] | null = null;
private albums: library.Album[] | null = null; private albums: library.Album[] | null = null;
private artists: library.Artist[] | null = null; private artists: library.Artist[] | null = null;
private genres: library.GenreWithCount[] | null = null; private genres: library.GenreWithCount[] | null = null;
@@ -91,8 +81,8 @@ class LibraryStore {
private cacheGen = 0; private cacheGen = 0;
constructor() { constructor() {
EventsOn(Events.LibraryScanComplete, (payload: unknown) => { EventsOn(Events.LibraryScanComplete, () => {
this.applyScanComplete(payload); this.invalidate();
}); });
EventsOn(Events.LibraryRemoved, () => { EventsOn(Events.LibraryRemoved, () => {
this.libraries = null; this.libraries = null;
@@ -108,8 +98,8 @@ class LibraryStore {
this.changeGen++; this.changeGen++;
this.notify(); this.notify();
}); });
EventsOn(Events.TrackMetadataChanged, (payload: unknown) => { EventsOn(Events.TrackMetadataChanged, () => {
this.applyMetadataChanged(payload); this.invalidate();
}); });
EventsOn(Events.TrackPlayCountChanged, (payload: unknown) => { EventsOn(Events.TrackPlayCountChanged, (payload: unknown) => {
this.applyPlayCount(payload); this.applyPlayCount(payload);
@@ -119,82 +109,34 @@ class LibraryStore {
}); });
this.loadCoverSize(); this.loadCoverSize();
this.warmSmallCollectionsOnIdle(); this.deferEagerFetch();
} }
/** /**
* Warm the three small collections once the app is idle after first * Schedules eagerFetch() to run after the DOM is ready.
* paint, and deliberately not the tracks (#280). * The LibraryStore singleton is instantiated during ES module
* * evaluation (import time), so calling eagerFetch() in the
* All four used to be fetched together at `DOMContentLoaded`, * constructor would fire 4 backend roundtrips before the app
* whichever view was showing. On a 26 138-track library the track * shell has rendered. Deferring to the 'DOMContentLoaded'
* list was 20.5 MB of that, and encoding it cost the backend ~170 MB * event (or calling immediately if the DOM is already parsed)
* of transient allocation — paid by someone looking at Home, which * lets the shell paint first, then begins data loading.
* needs none of it. Measured on 50 000 tracks: 543 MB of backend RSS
* at rest before, 296 MB after, and 12.1 MB of binding bytes instead
* of 35.9.
*
* Albums, artists and genres are 1.6 MB together, so they are still
* fetched ahead of the click — that is what made those views
* instant. Tracks are 12 MB at 50 000 and are fetched by the view
* that draws them, or by `prefetch` from a hover.
*
* Idle rather than immediate: this is speculative, so it must not
* compete with the first paint.
*/ */
private warmSmallCollectionsOnIdle(): void { private deferEagerFetch(): void {
const warm = () => { if (document.readyState === 'loading') {
const logged = this.failureReporter(); window.addEventListener(
'DOMContentLoaded',
void this.getAlbums().catch(logged('albums')); () => {
void this.getArtists().catch(logged('artists')); this.eagerFetch();
void this.getGenres().catch(logged('genres')); },
}; { once: true },
);
if (typeof requestIdleCallback === 'function') {
requestIdleCallback(warm, { timeout: WARM_IDLE_TIMEOUT_MS });
} else { } else {
setTimeout(warm, 0); // DOM already parsed (shouldn't happen during module
// eval, but handles dynamic instantiation safely).
this.eagerFetch();
} }
} }
/**
* Start loading what a view will need, without waiting for it.
*
* A nav item calls this on hover or focus: that is the ~100 ms
* before the click, and it is what #280 trades for not paying for
* every collection at startup whether or not anyone goes there.
*
* A view this store holds nothing for is ignored rather than an
* error — it is a hint, and a hint about Home is not a mistake.
* The failure is swallowed here because the view that wanted the
* data reports it: this is the same request, already deduplicated
* by `inFlight`, and it has no caller to reject to.
*/
prefetch(view: string): void {
const started = (() => {
switch (view) {
case 'tracks':
return this.getTracks();
case 'albums':
return this.getAlbums();
case 'artists':
return this.getArtists();
case 'genres':
return this.getGenres();
default:
return null;
}
})();
void started?.catch(() => undefined);
}
private failureReporter(): (what: string) => (err: unknown) => void {
return (what) => (err) =>
console.error(`library: could not load ${what}`, err);
}
// =================================================================== // ===================================================================
// DATA ACCESS // DATA ACCESS
// Returns cached data or fetches from backend on first access. // Returns cached data or fetches from backend on first access.
@@ -272,18 +214,18 @@ class LibraryStore {
} }
} }
async getTracks(): Promise<ListTrack[]> { async getTracks(): Promise<library.Track[]> {
if (this.tracks !== null) { if (this.tracks !== null) {
return this.tracks; return this.tracks;
} }
const pending = this.pending<ListTrack[]>('tracks'); const pending = this.pending<library.Track[]>('tracks');
if (pending) return pending; if (pending) return pending;
return this.track( return this.track(
'tracks', 'tracks',
GetTrackTable(this.libraryFilter()).then(decodeTrackTable), list(GetTracks(this.libraryFilter())),
(tracks) => { (tracks) => {
this.tracks = tracks; this.tracks = tracks;
}, },
@@ -399,7 +341,7 @@ class LibraryStore {
// Synchronous access for controllers that need current cached values. // Synchronous access for controllers that need current cached values.
// =================================================================== // ===================================================================
getCachedTracks(): ListTrack[] | null { getCachedTracks(): library.Track[] | null {
return this.tracks; return this.tracks;
} }
@@ -555,116 +497,6 @@ class LibraryStore {
// INVALIDATION // INVALIDATION
// =================================================================== // ===================================================================
/**
* A scan finished. Reload only if it changed something (#282).
*
* `ScanMetrics` says how many files were added, updated and removed,
* and a soft rescan of an unchanged library reports zero of each —
* yet every loaded collection was thrown away and refetched, which
* at 26 138 tracks is 20.5 MB across the IPC for a scan that found
* nothing. Anything non-zero still reloads everything: a scan can
* change a tag, an album name or a genre on any file it touched, and
* it reports counts rather than paths.
*
* A payload that says nothing at all is treated as a change, not as
* a no-op: an unknown shape must not be able to leave a stale list
* on screen.
*/
private applyScanComplete(payload: unknown): void {
const m = payload as {
added?: number;
updated?: number;
removed?: number;
} | null;
if (!m) {
this.invalidate();
return;
}
const changed = (m.added ?? 0) + (m.updated ?? 0) + (m.removed ?? 0);
if (changed === 0) return;
this.invalidate();
}
/**
* Tags were rewritten on disk. A single file is patched; a batch is
* a reload (#282).
*
* The event names the one file it rewrote, and that file's row is
* the only row that changed — so re-reading the whole list to pick
* up one new title is 20.5 MB at 26 138 tracks. A batch write
* carries no paths (it can be thousands of files), and the summaries
* really do change with it, so that one still reloads.
*/
private applyMetadataChanged(payload: unknown): void {
const p = payload as { filePath?: string; batch?: boolean } | null;
if (!p?.filePath || p.batch) {
this.invalidate();
return;
}
this.patchOneTrack(p.filePath);
}
/**
* Re-read one file's row and splice it in.
*
* The summaries are dropped and refetched, because a retag can move
* a track between albums and change a genre — they are the small
* collections, and they are what makes the album and artist views
* agree with the row that was just patched.
*/
private patchOneTrack(filePath: string): void {
const loaded = this.loadedCollections();
void trackCache
.refresh([filePath])
.then(([fresh]) => {
if (!fresh || this.tracks === null) return;
const idx = this.tracks.findIndex(
(t) => t.FilePath === filePath,
);
// Not in this view's list (another library's file, or
// gone): the list is right as it stands.
if (idx === -1) return;
this.tracks = [
...this.tracks.slice(0, idx),
fresh,
...this.tracks.slice(idx + 1),
];
this.changeGen++;
this.notify();
})
.catch((err: unknown) => {
// The patch failed, so the list may now be stale. Fall
// back to the answer that cannot be wrong.
console.error('library: could not re-read a retagged track', err);
this.invalidate();
});
this.albums = null;
this.artists = null;
this.genres = null;
this.cacheGen++;
this.inFlight.delete('albums');
this.inFlight.delete('artists');
this.inFlight.delete('genres');
this.refetchLoaded({
albums: loaded.albums,
artists: loaded.artists,
genres: loaded.genres,
});
}
/** /**
* Patch one track's play statistics in place. * Patch one track's play statistics in place.
* *
@@ -688,11 +520,10 @@ class LibraryStore {
private applyPlayCount(payload: unknown): void { private applyPlayCount(payload: unknown): void {
if (this.tracks === null) return; if (this.tracks === null) return;
// The list does not carry LastPlayed (#281); trackCache patches
// it on the whole tracks the details dialog reads.
const p = payload as { const p = payload as {
filePath?: string; filePath?: string;
playCount?: number; playCount?: number;
lastPlayed?: string;
} | null; } | null;
if (!p?.filePath) return; if (!p?.filePath) return;
@@ -705,10 +536,14 @@ class LibraryStore {
if (existing === undefined) return; if (existing === undefined) return;
const patched: ListTrack = { const patched = Object.assign(
...existing, Object.create(Object.getPrototypeOf(existing) as object),
existing,
{
PlayCount: p.playCount ?? existing.PlayCount, PlayCount: p.playCount ?? existing.PlayCount,
}; LastPlayed: p.lastPlayed ?? existing.LastPlayed,
},
) as library.Track;
this.tracks = [ this.tracks = [
...this.tracks.slice(0, idx), ...this.tracks.slice(0, idx),
@@ -762,8 +597,6 @@ class LibraryStore {
} }
} }
const loaded = this.loadedCollections();
this.albums = null; this.albums = null;
this.artists = null; this.artists = null;
this.genres = null; this.genres = null;
@@ -779,63 +612,50 @@ class LibraryStore {
this.changeGen++; this.changeGen++;
this.notify(); this.notify();
// Only the summaries something was showing (#280): a removal const logged = (what: string) => (err: unknown) =>
// nobody was looking at does not load a collection to correct it. console.error(`library: could not reload ${what}`, err);
this.refetchLoaded({
albums: loaded.albums, void this.getAlbums().catch(logged('albums'));
artists: loaded.artists, void this.getArtists().catch(logged('artists'));
genres: loaded.genres, void this.getGenres().catch(logged('genres'));
});
} }
private invalidate(): void { private invalidate(): void {
// Which collections something has actually loaded, before they
// are dropped. Refetching all four here would undo #280: a scan
// finishing would fetch the track list of a library nobody has
// opened the Tracks view on.
const loaded = this.loadedCollections();
this.tracks = null; this.tracks = null;
this.albums = null; this.albums = null;
this.artists = null; this.artists = null;
this.genres = null; this.genres = null;
// Anything still in flight was asked for on behalf of a // Anything still in flight was asked for on behalf of a
// selection that no longer applies: forget it, so the refetch // selection that no longer applies: forget it, so the eager
// below starts a request for the current one rather than // refetch below starts a request for the current one rather
// adopting the old one's answer. // than adopting the old one's answer.
this.inFlight.clear(); this.inFlight.clear();
this.cacheGen++; this.cacheGen++;
this.changeGen++; this.changeGen++;
this.scrollPositions = { tracks: 0, albums: 0, artists: 0, genres: 0 }; this.scrollPositions = { tracks: 0, albums: 0, artists: 0, genres: 0 };
this.notify(); this.notify();
this.refetchLoaded(loaded); this.eagerFetch();
}
/** The collections currently held, keyed by the view that draws them. */
private loadedCollections(): Partial<Record<ViewName, boolean>> {
return {
tracks: this.tracks !== null,
albums: this.albums !== null,
artists: this.artists !== null,
genres: this.genres !== null,
};
} }
/** /**
* Refetch exactly the collections named, and nothing else. * Fetches all library data. Called after DOM ready
* * (initial load, via deferEagerFetch) and after cache
* A failed fetch is reported by whichever view asked for the data * invalidation so that controller subscribers receive
* (it is that panel's failure, not the app's), but this refetch has * fresh data on the next requestUpdate() cycle without
* no caller to reject to — without a catch it is an unhandled * needing their own LibraryScanComplete listener.
* rejection.
*/ */
private refetchLoaded(loaded: Partial<Record<ViewName, boolean>>): void { private eagerFetch(): void {
const logged = this.failureReporter(); // A failed fetch is reported by whichever view asked for the
// data (it is that panel's failure, not the app's), but the
// eager refetch has no caller to reject to — without a catch it
// is an unhandled rejection.
const logged = (what: string) => (err: unknown) =>
console.error(`library: could not load ${what}`, err);
if (loaded.tracks) void this.getTracks().catch(logged('tracks')); void this.getTracks().catch(logged('tracks'));
if (loaded.albums) void this.getAlbums().catch(logged('albums')); void this.getAlbums().catch(logged('albums'));
if (loaded.artists) void this.getArtists().catch(logged('artists')); void this.getArtists().catch(logged('artists'));
if (loaded.genres) void this.getGenres().catch(logged('genres')); void this.getGenres().catch(logged('genres'));
} }
// =================================================================== // ===================================================================
-240
View File
@@ -1,240 +0,0 @@
/**
* Whole tracks, looked up by file path, for the rows a surface is
* actually showing (#279).
*
* Track details from the queue, a playlist or a smart playlist used to
* find their track in `libraryStore`'s whole-library array. That made
* the array a dependency of every surface that can open details, so it
* was fetched eagerly at startup — 20.5 MB of JSON at 26 138 tracks —
* and when it had *not* landed yet, the openers that read it
* synchronously did nothing at all. This asks the backend for the
* paths in hand instead.
*
* Three things are load-bearing, the same three as `credit-store`:
*
* **Lookups are coalesced.** Every `get()` made in the same task joins
* one `GetTracksByPaths` call. A timer rather than a frame: these are
* user actions, not row renders, and a frame never fires in a hidden
* window, which would leave the caller waiting on nothing.
*
* **It is bounded.** A batch details dialog over "Select all" asks for
* every track in the library; it gets every one of them back, but the
* cache keeps only the most recent `TRACK_CACHE_LIMIT`.
*
* **It forgets what changed.** The events that make `library-store`
* refetch drop the affected entries here, and a play count is patched
* in place, so a cached track is never older than the last event about
* it. An answer that was in flight across an invalidation is still
* returned to its caller (it was correct when asked) but not cached.
*/
import { EventsOn } from '@runtime/runtime';
import { GetTracksByPaths } from '@go/library/library.js';
import type * as library from '@go/library/models.js';
import { list } from '@utils/binding';
import { LRUMap } from '@utils/lru-map';
import { registerCacheProbe } from '@utils/cache-stats';
import { Events } from '../events';
/**
* Tracks retained. A track is ~25 short fields, so this is a couple of
* megabytes at most — sized above any batch a person edits by hand,
* far below a library.
*/
export const TRACK_CACHE_LIMIT = 2_000;
interface Waiter {
paths: readonly string[];
resolve: (tracks: library.Track[]) => void;
reject: (err: unknown) => void;
}
class TrackCache {
private cache = new LRUMap<string, library.Track>(TRACK_CACHE_LIMIT);
/** Requests collected in this task, answered by one binding call. */
private waiting: Waiter[] = [];
private flushHandle: ReturnType<typeof setTimeout> | null = null;
/** Bumped by every invalidation; an older answer is not cached. */
private gen = 0;
constructor() {
EventsOn(Events.LibraryScanComplete, () => this.clear());
EventsOn(Events.LibraryRemoved, () => this.clear());
EventsOn(Events.TrackMetadataChanged, (payload: unknown) => {
const p = payload as { filePath?: string } | null;
// A batch write names no paths: forget everything.
if (p?.filePath) this.forget([p.filePath]);
else this.clear();
});
EventsOn(Events.TracksRemovedFromLibrary, (payload: unknown) => {
const p = payload as { filePaths?: string[] } | null;
this.forget(p?.filePaths ?? []);
});
EventsOn(Events.TrackPlayCountChanged, (payload: unknown) => {
this.applyPlayCount(payload);
});
registerCacheProbe('tracks-by-path', () => ({
entries: this.cache.size,
chars: this.retainedChars(),
limit: TRACK_CACHE_LIMIT,
}));
}
/**
* The tracks at `paths`, in the order given, without the ones that
* are not in the library. Cached tracks are answered without a call.
*/
get(paths: readonly string[]): Promise<library.Track[]> {
const answered = this.fromCache(paths);
if (answered) return Promise.resolve(answered);
return new Promise((resolve, reject) => {
this.waiting.push({ paths, resolve, reject });
this.flushHandle ??= setTimeout(() => void this.flush(), 0);
});
}
/** One track, or undefined when the path is not in the library. */
async getOne(path: string): Promise<library.Track | undefined> {
return (await this.get([path]))[0];
}
/**
* Ask the backend again for `paths`, ignoring the cache.
*
* For a caller that has just written to these files and must not be
* answered from before the write, whether or not the event that
* invalidates them has arrived yet.
*/
refresh(paths: readonly string[]): Promise<library.Track[]> {
this.forget(paths);
return this.get(paths);
}
/** Every path cached, or null if any is missing. */
private fromCache(paths: readonly string[]): library.Track[] | null {
const tracks: library.Track[] = [];
for (const path of paths) {
const track = this.cache.get(path);
if (!track) return null;
tracks.push(track);
}
return tracks;
}
private async flush(): Promise<void> {
this.flushHandle = null;
const waiting = this.waiting;
this.waiting = [];
const missing = new Set<string>();
for (const w of waiting) {
for (const path of w.paths) {
if (!this.cache.has(path)) missing.add(path);
}
}
const gen = this.gen;
let fetched: library.Track[];
try {
fetched = missing.size > 0
? await list(GetTracksByPaths([...missing]))
: [];
} catch (err) {
for (const w of waiting) w.reject(err);
return;
}
// Answer from what this call returned plus what was cached when
// it was made — not from the cache afterwards, which an LRU
// eviction or an invalidation may have emptied in between.
const answer = new Map<string, library.Track>();
for (const w of waiting) {
for (const path of w.paths) {
const hit = this.cache.get(path);
if (hit) answer.set(path, hit);
}
}
for (const track of fetched) {
answer.set(track.FilePath, track);
if (gen === this.gen) this.cache.set(track.FilePath, track);
}
for (const w of waiting) {
const tracks: library.Track[] = [];
for (const path of w.paths) {
const track = answer.get(path);
if (track) tracks.push(track);
}
w.resolve(tracks);
}
}
private forget(paths: readonly string[]): void {
for (const path of paths) this.cache.delete(path);
this.gen++;
}
private clear(): void {
this.cache.clear();
this.gen++;
}
private applyPlayCount(payload: unknown): void {
const p = payload as {
filePath?: string;
playCount?: number;
lastPlayed?: string;
} | null;
if (!p?.filePath) return;
const existing = this.cache.get(p.filePath);
if (!existing) return;
this.cache.set(p.filePath, {
...existing,
PlayCount: p.playCount ?? existing.PlayCount,
LastPlayed: p.lastPlayed ?? existing.LastPlayed,
});
}
private retainedChars(): number {
let total = 0;
for (const t of this.cache.values()) {
total += t.FilePath.length + t.TrackName.length
+ t.ArtistName.length + t.Album.length;
}
return total;
}
}
export const trackCache = new TrackCache();
-8
View File
@@ -79,14 +79,6 @@ export function compact<V>(
return out; return out;
} }
/**
* listField is list for a slice that arrived as a *field* of a struct
* rather than as a return value — a column of `TrackTable`, say.
*/
export function listField<T>(field: T[] | null | undefined): T[] {
return field ?? [];
}
/** /**
* value awaits a binding whose result is used as-is, dropping only the * value awaits a binding whose result is used as-is, dropping only the
* cancellation the app never asks for. * cancellation the app never asks for.
+25 -94
View File
@@ -1,70 +1,40 @@
/** /**
* Open `<track-details>` for a file path. * Open `<track-details>` for a file path.
* *
* Explore's rows carry no library metadata at all: a tracklist row is * The five library-side hosts already hold the `library.Track` the
* an `MBTrack`/`LBTopRecording` from the catalog, and all it can say * dialog wants — they render it. Explore's rows do not: a tracklist row
* is an `MBTrack`/`LBTopRecording` from the catalog, and all it can say
* about the library is *which file is behind it*. So the path is the * about the library is *which file is behind it*. So the path is the
* one key both sides share, and turning it back into a track is the * one key both sides share, and turning it back into a track is the
* work this does. The queue, playlists and smart playlists are the same * work this does.
* shape — they render their own row type.
* *
* All of them used to find the track in `libraryStore`'s whole-library * `libraryStore.getTracks()` is awaited rather than
* array, which meant that array had to be loaded for details to open — * `getCachedTracks()`-and-bail (which is what `queue-panel` does):
* and the ones that read it synchronously silently did nothing when it * Explore is reachable without ever opening the library views, so a
* was not (#279). They ask `trackCache` for the paths in hand instead: * cold cache is ordinary here rather than a symptom, and silently doing
* one small call, coalesced, answered from cache the second time. * nothing on a menu item the user just clicked is not an option. The
* * fetch is the store's own, shared with every other reader.
* A caller that *is* showing the rows passes them
* (`showBatchTrackDetails`), because fetching back what is already in
* hand is how a "select all" over 50 000 tracks came to cost 3 s.
*/ */
import type * as library from '@go/library/models.js';
import type { import type {
CoverArtUrls, CoverArtUrls,
TrackDetails, TrackDetails,
} from '@components/track-details/track-details.js'; } from '@components/track-details/track-details.js';
import { trackCache } from '@store/track-cache.js'; import { libraryStore } from '@store/library-store.js';
import { loadTrackDetails } from '@utils/lazy-track-details.js'; import { loadTrackDetails } from '@utils/lazy-track-details.js';
import type * as library from '@go/library/models.js'; import { tracksByFilePath } from '@utils/track-index.js';
import type { ListTrack } from '@utils/track-table';
/** /** The cover art the dialog shows, or nothing when the track has none. */
* The cover art the dialog shows, or nothing when the track has none. function coverArtOf(track: library.Track): CoverArtUrls | undefined {
* return track.CoverArtPath
* A whole track carries every tier; the list's rows carry only the ? {
* small one (#281), and every tier is derived from the same file — so a coverArtPath: track.CoverArtPath,
* missing one falls back to the tier that is there rather than the coverArtSmall: track.CoverArtSmall,
* dialog showing nothing. coverArtMedium: track.CoverArtMedium,
*/ coverArtLarge: track.CoverArtLarge,
function coverArtOf(track: ListTrack): CoverArtUrls | undefined { }
const small = track.CoverArtSmall; : undefined;
if (!small) return undefined;
const tiers = track as Partial<library.Track>;
return {
coverArtPath: tiers.CoverArtPath ?? small,
coverArtSmall: small,
coverArtMedium: tiers.CoverArtMedium ?? small,
coverArtLarge: tiers.CoverArtLarge ?? small,
};
}
/**
* The cover a batch dialog shows: the album's, when every track is on
* one album; none and `mixed` when they span several.
*/
export function batchCoverArt(
tracks: readonly ListTrack[],
): { coverArt: CoverArtUrls | null; mixed: boolean } {
const albums = new Set(tracks.map((t) => t.Album));
if (albums.size > 1) return { coverArt: null, mixed: true };
const first = tracks[0];
return { coverArt: first ? coverArtOf(first) ?? null : null, mixed: false };
} }
/** /**
@@ -92,7 +62,8 @@ export async function showTrackDetailsForPath(
filePath: string, filePath: string,
retry: () => void, retry: () => void,
): Promise<TrackDetailsOutcome> { ): Promise<TrackDetailsOutcome> {
const track = await trackCache.getOne(filePath); const tracks = await libraryStore.getTracks();
const track = tracksByFilePath(tracks).get(filePath);
if (!track) return 'not-in-library'; if (!track) return 'not-in-library';
@@ -104,43 +75,3 @@ export async function showTrackDetailsForPath(
return 'shown'; return 'shown';
} }
/**
* Show the batch details dialog for the library tracks at `filePaths`,
* in that order. Paths not in the library are left out; when none are,
* nothing is shown.
*/
export async function showBatchTrackDetailsForPaths(
dialog: () => TrackDetails | undefined,
filePaths: readonly string[],
retry: () => void,
): Promise<TrackDetailsOutcome> {
return showBatchTrackDetails(dialog, await trackCache.get(filePaths), retry);
}
/**
* Show the batch details dialog for tracks the caller already holds.
*
* The Tracks view is why this exists rather than only the path form: a
* selection there can be the whole library, and fetching every track of
* it back to read fields the rows in hand already carry cost 3 s and
* 700 ms of blocked main thread on 50 000 tracks (#281's own regression,
* measured).
*/
export async function showBatchTrackDetails(
dialog: () => TrackDetails | undefined,
tracks: readonly ListTrack[],
retry: () => void,
): Promise<TrackDetailsOutcome> {
if (tracks.length === 0) return 'not-in-library';
const ready = await loadTrackDetails(retry);
if (!ready) return 'chunk-failed';
const { coverArt, mixed } = batchCoverArt(tracks);
dialog()?.showBatch(tracks, coverArt, mixed);
return 'shown';
}
+14 -14
View File
@@ -20,22 +20,22 @@
* given array and never again. * given array and never again.
*/ */
/** Anything keyed by file path: a whole track, or the list's row. */ import type * as library from '@go/library/models.js';
interface HasFilePath {
FilePath: string;
}
const byArray = new WeakMap<readonly HasFilePath[], Map<string, HasFilePath>>(); const byArray = new WeakMap<
readonly library.Track[],
Map<string, library.Track>
>();
/** The lookup for `tracks`, built once per array identity. */ /** The lookup for `tracks`, built once per array identity. */
export function tracksByFilePath<T extends HasFilePath>( export function tracksByFilePath(
tracks: readonly T[], tracks: readonly library.Track[],
): Map<string, T> { ): Map<string, library.Track> {
let map = byArray.get(tracks) as Map<string, T> | undefined; let map = byArray.get(tracks);
if (map) return map; if (map) return map;
map = new Map<string, T>(); map = new Map<string, library.Track>();
for (const track of tracks) { for (const track of tracks) {
// First wins: a duplicate path would be the same file, and // First wins: a duplicate path would be the same file, and
@@ -49,12 +49,12 @@ export function tracksByFilePath<T extends HasFilePath>(
} }
/** Resolve file paths to tracks, dropping any that are not present. */ /** Resolve file paths to tracks, dropping any that are not present. */
export function tracksForPaths<T extends HasFilePath>( export function tracksForPaths(
tracks: readonly T[], tracks: readonly library.Track[],
filePaths: readonly string[], filePaths: readonly string[],
): T[] { ): library.Track[] {
const byPath = tracksByFilePath(tracks); const byPath = tracksByFilePath(tracks);
const result: T[] = []; const result: library.Track[] = [];
for (const filePath of filePaths) { for (const filePath of filePaths) {
const track = byPath.get(filePath); const track = byPath.get(filePath);
-128
View File
@@ -1,128 +0,0 @@
/**
* Decode the backend's `TrackTable` into the rows the Tracks view uses.
*
* The table is one array per column with every repeated string sent
* once (#281): ~167 bytes a track against the ~800 of the object-per-
* track JSON it replaced. This is its only decoder, and
* `backend/library/tracktable.go` its only encoder.
*
* Decoding builds plain objects of the same shape as before, so the
* filter, sort and selection code did not have to change — but every
* occurrence of a repeated string is now the *same* string, and every
* track with one genre list shares the array, which is part of why the
* JS heap shrinks along with the payload. Shared means read-only: no
* caller mutates a row's `Genre`.
*/
import type * as library from '@go/library/models.js';
import { listField } from '@utils/binding';
/**
* A track as the list carries it. The fields left out are the details
* dialog's, which reads whole tracks by path from `trackCache`.
*/
export type ListTrack = Omit<
library.Track,
'LastPlayed' | 'CoverArtPath' | 'CoverArtMedium' | 'CoverArtLarge'
>;
/** The `Track` fields a `ListTrack` does not carry. */
export const LIST_TRACK_OMITS = [
'LastPlayed',
'CoverArtPath',
'CoverArtMedium',
'CoverArtLarge',
] as const;
/** A table whose columns disagree in length is a broken encoder, not data. */
export class TrackTableError extends Error {}
export function decodeTrackTable(table: library.TrackTable): ListTrack[] {
const strings = listField(table.strings);
const filePath = listField(table.filePath);
const n = filePath.length;
const str = (col: number[] | null, name: string): string[] => {
const idx = listField(col);
if (idx.length !== n) throw mismatch(name, idx.length, n);
return idx.map((i) => {
const s = strings[i];
if (s === undefined) throw new TrackTableError(`${name}: string ${i} out of range`);
return s;
});
};
const num = (col: number[] | null, name: string): number[] => {
const v = listField(col);
if (v.length !== n) throw mismatch(name, v.length, n);
return v;
};
const genreSets = listField(table.genreSets).map((set) =>
listField(set).map((i) => strings[i] ?? ''),
);
const genre = num(table.genre, 'genre');
const trackName = str(table.trackName, 'trackName');
const artistName = str(table.artistName, 'artistName');
const album = str(table.album, 'album');
const composer = str(table.composer, 'composer');
const fileType = str(table.fileType, 'fileType');
const artistMbid = str(table.artistMbid, 'artistMbid');
const releaseGroupMbid = str(table.releaseGroupMbid, 'releaseGroupMbid');
const recordingMbid = str(table.recordingMbid, 'recordingMbid');
const coverArtSmall = str(table.coverArtSmall, 'coverArtSmall');
const lengthMs = num(table.lengthMs, 'lengthMs');
const trackNumber = num(table.trackNumber, 'trackNumber');
const discNumber = num(table.discNumber, 'discNumber');
const year = num(table.year, 'year');
const sampleRate = num(table.sampleRate, 'sampleRate');
const bitDepth = num(table.bitDepth, 'bitDepth');
const channels = num(table.channels, 'channels');
const bitrate = num(table.bitrate, 'bitrate');
const fileSize = num(table.fileSize, 'fileSize');
const playCount = num(table.playCount, 'playCount');
const rows: ListTrack[] = new Array<ListTrack>(n);
for (let i = 0; i < n; i++) {
const g = genreSets[genre[i]!];
if (g === undefined) throw new TrackTableError(`genre: set ${genre[i]} out of range`);
rows[i] = {
FilePath: filePath[i]!,
TrackName: trackName[i]!,
ArtistName: artistName[i]!,
Album: album[i]!,
Composer: composer[i]!,
FileType: fileType[i]!,
Genre: g,
ArtistMBID: artistMbid[i]!,
ReleaseGroupMBID: releaseGroupMbid[i]!,
RecordingMBID: recordingMbid[i]!,
CoverArtSmall: coverArtSmall[i]!,
TrackLength: String(lengthMs[i]!),
TrackNumber: trackNumber[i]!,
DiscNumber: discNumber[i]!,
Year: year[i]!,
SampleRate: sampleRate[i]!,
BitDepth: bitDepth[i]!,
Channels: channels[i]!,
Bitrate: bitrate[i]!,
FileSize: fileSize[i]!,
PlayCount: playCount[i]!,
};
}
return rows;
}
function mismatch(name: string, got: number, want: number): TrackTableError {
return new TrackTableError(`${name}: ${got} values for ${want} tracks`);
}
@@ -19,7 +19,6 @@ import '@components/cover-grid/cover-grid';
import { emit, stub, flush, resetHarness } from '@test/support/harness'; import { emit, stub, flush, resetHarness } from '@test/support/harness';
import { Events } from '../../src/events'; import { Events } from '../../src/events';
import { fixture, shadowAll } from '@test/support/render'; import { fixture, shadowAll } from '@test/support/render';
import { trackTable } from '@test/support/track-table';
const LONG = const LONG =
'The Rise and Fall of a Midwest Princess in the Key of Everything'; 'The Rise and Fall of a Midwest Princess in the Key of Everything';
@@ -50,7 +49,7 @@ describe('the album card’s year', () => {
beforeEach(() => { beforeEach(() => {
resetHarness(); resetHarness();
stub('library.Library.GetAlbums', ALBUMS); stub('library.Library.GetAlbums', ALBUMS);
stub('library.Library.GetTrackTable', trackTable([])); stub('library.Library.GetTracks', []);
emit(Events.LibraryScanComplete); emit(Events.LibraryScanComplete);
}); });
@@ -22,7 +22,6 @@ import '@components/cover-grid/cover-grid';
import { emit, stub, flush, resetHarness } from '@test/support/harness'; import { emit, stub, flush, resetHarness } from '@test/support/harness';
import { Events } from '../../src/events'; import { Events } from '../../src/events';
import { fixture, shadow, shadowAll } from '@test/support/render'; import { fixture, shadow, shadowAll } from '@test/support/render';
import { trackTable } from '@test/support/track-table';
/** /**
* Enough albums to fill more than one row. * Enough albums to fill more than one row.
@@ -83,7 +82,7 @@ describe('the album dropdown', () => {
beforeEach(() => { beforeEach(() => {
resetHarness(); resetHarness();
stub('library.Library.GetAlbums', ALBUMS); stub('library.Library.GetAlbums', ALBUMS);
stub('library.Library.GetTrackTable', trackTable([])); stub('library.Library.GetTracks', []);
stub('library.Library.GetAlbumTracks', TRACKS); stub('library.Library.GetAlbumTracks', TRACKS);
stub('library.Library.GetAlbumTracks', TRACKS); stub('library.Library.GetAlbumTracks', TRACKS);
emit(Events.LibraryScanComplete); emit(Events.LibraryScanComplete);
@@ -156,7 +155,7 @@ describe('the albums grid scrolls', () => {
beforeEach(() => { beforeEach(() => {
resetHarness(); resetHarness();
stub('library.Library.GetAlbums', ALBUMS); stub('library.Library.GetAlbums', ALBUMS);
stub('library.Library.GetTrackTable', trackTable([])); stub('library.Library.GetTracks', []);
emit(Events.LibraryScanComplete); emit(Events.LibraryScanComplete);
}); });
+5 -6
View File
@@ -20,7 +20,6 @@ import { emit, stub, flush, resetHarness } from '@test/support/harness';
import { Events } from '../../src/events'; import { Events } from '../../src/events';
import { fixture, shadow, shadowAll } from '@test/support/render'; import { fixture, shadow, shadowAll } from '@test/support/render';
import { searchStore } from '@store/search-store'; import { searchStore } from '@store/search-store';
import { trackTable } from '@test/support/track-table';
/** /**
* The searchable columns' accessors read these fields and call * The searchable columns' accessors read these fields and call
@@ -77,7 +76,7 @@ describe('the track list says how it is sorted', () => {
beforeEach(async () => { beforeEach(async () => {
resetHarness(); resetHarness();
searchStore.setTerm(''); searchStore.setTerm('');
stub('library.Library.GetTrackTable', trackTable(TRACKS)); stub('library.Library.GetTracks', TRACKS);
stub('library.Library.GetAlbums', []); stub('library.Library.GetAlbums', []);
emit(Events.LibraryScanComplete); emit(Events.LibraryScanComplete);
}); });
@@ -133,7 +132,7 @@ describe('the track list has a voice for its own state', () => {
}); });
it('announces the result of a search that matches nothing', async () => { it('announces the result of a search that matches nothing', async () => {
stub('library.Library.GetTrackTable', trackTable(TRACKS)); stub('library.Library.GetTracks', TRACKS);
emit(Events.LibraryScanComplete); emit(Events.LibraryScanComplete);
const el = await fixture<LitElement>('track-list'); const el = await fixture<LitElement>('track-list');
@@ -164,7 +163,7 @@ describe('a selectable grid is a listbox, not a row of buttons', () => {
searchStore.setTerm(''); searchStore.setTerm('');
stub('library.Library.GetArtists', ARTISTS); stub('library.Library.GetArtists', ARTISTS);
stub('library.Library.GetGenres', GENRES); stub('library.Library.GetGenres', GENRES);
stub('library.Library.GetTrackTable', trackTable([])); stub('library.Library.GetTracks', []);
stub('library.Library.GetAlbums', []); stub('library.Library.GetAlbums', []);
emit(Events.LibraryScanComplete); emit(Events.LibraryScanComplete);
}); });
@@ -198,7 +197,7 @@ describe('a clipped value is readable somewhere', () => {
beforeEach(async () => { beforeEach(async () => {
resetHarness(); resetHarness();
searchStore.setTerm(''); searchStore.setTerm('');
stub('library.Library.GetTrackTable', trackTable(TRACKS)); stub('library.Library.GetTracks', TRACKS);
stub('library.Library.GetAlbums', []); stub('library.Library.GetAlbums', []);
emit(Events.LibraryScanComplete); emit(Events.LibraryScanComplete);
}); });
@@ -241,7 +240,7 @@ describe('the playing row is more than a colour', () => {
beforeEach(async () => { beforeEach(async () => {
resetHarness(); resetHarness();
searchStore.setTerm(''); searchStore.setTerm('');
stub('library.Library.GetTrackTable', trackTable(TRACKS)); stub('library.Library.GetTracks', TRACKS);
stub('library.Library.GetAlbums', []); stub('library.Library.GetAlbums', []);
emit(Events.LibraryScanComplete); emit(Events.LibraryScanComplete);
}); });
@@ -35,7 +35,6 @@ import {
imagePrefetched, imagePrefetched,
resetImagePrefetch, resetImagePrefetch,
} from '@utils/image-prefetch'; } from '@utils/image-prefetch';
import { trackTable } from '@test/support/track-table';
/** Enough albums that the virtualizer's own window is nowhere near the end. */ /** Enough albums that the virtualizer's own window is nowhere near the end. */
const ALBUMS = Array.from({ length: 400 }, (_, i) => { const ALBUMS = Array.from({ length: 400 }, (_, i) => {
@@ -119,7 +118,7 @@ beforeEach(() => {
localStorage.clear(); localStorage.clear();
stub('library.Library.GetAlbums', ALBUMS); stub('library.Library.GetAlbums', ALBUMS);
stub('library.Library.GetArtists', ARTISTS); stub('library.Library.GetArtists', ARTISTS);
stub('library.Library.GetTrackTable', trackTable([])); stub('library.Library.GetTracks', []);
stub('library.Library.GetGenres', []); stub('library.Library.GetGenres', []);
emit(Events.LibraryScanComplete); emit(Events.LibraryScanComplete);
}); });
@@ -29,7 +29,6 @@ import '@components/genres-view/genres-view';
import { emit, stub, flush, resetHarness } from '@test/support/harness'; import { emit, stub, flush, resetHarness } from '@test/support/harness';
import { Events } from '../../src/events'; import { Events } from '../../src/events';
import { fixture, shadowAll } from '@test/support/render'; import { fixture, shadowAll } from '@test/support/render';
import { trackTable } from '@test/support/track-table';
const ARTISTS = [ const ARTISTS = [
{ ID: 1, Name: 'Alpha', AlbumCount: 2, TrackCount: 9 }, { ID: 1, Name: 'Alpha', AlbumCount: 2, TrackCount: 9 },
@@ -59,7 +58,7 @@ describe('a card grid shows its selection', () => {
resetHarness(); resetHarness();
stub('library.Library.GetArtists', ARTISTS); stub('library.Library.GetArtists', ARTISTS);
stub('library.Library.GetGenres', GENRES); stub('library.Library.GetGenres', GENRES);
stub('library.Library.GetTrackTable', trackTable([])); stub('library.Library.GetTracks', []);
stub('library.Library.GetAlbums', []); stub('library.Library.GetAlbums', []);
// The views read through LibraryController, whose cache is only // The views read through LibraryController, whose cache is only
// primed by a scan-complete; without it they render nothing and the // primed by a scan-complete; without it they render nothing and the
+2 -11
View File
@@ -11,8 +11,7 @@ import '@components/library-filter/library-filter';
import '@components/library-status-indicator/library-status-indicator'; import '@components/library-status-indicator/library-status-indicator';
import { Events } from '../../src/events'; import { Events } from '../../src/events';
import { activeViewStore } from '@store/active-view-store'; import { activeViewStore } from '@store/active-view-store';
import { libraryStore } from '@store/library-store'; import { emit, stub, flush, calls, lastArgs } from '@test/support/harness';
import { emit, stub, flush, calls, lastArgs, resetHarness } from '@test/support/harness';
import { import {
fixture, fixture,
shadow, shadow,
@@ -182,14 +181,6 @@ describe('<library-filter>', () => {
it('selects a library by id, and the merged view by empty string', async () => { it('selects a library by id, and the merged view by empty string', async () => {
const el = await fixture('library-filter'); const el = await fixture('library-filter');
// The list has to be in use for the filter change to refetch it
// (#280): a collection nothing has loaded is not loaded to correct
// it, so this is the state the app is in when the Tracks view is up.
stub('library.Library.GetTrackTable', { strings: [''] });
await libraryStore.getTracks();
resetHarness();
stub('library.Library.GetTrackTable', { strings: [''] });
await flush(); await flush();
await el.updateComplete; await el.updateComplete;
@@ -200,7 +191,7 @@ describe('<library-filter>', () => {
select?.dispatchEvent(new Event('change')); select?.dispatchEvent(new Event('change'));
await flush(); await flush();
expect(lastArgs('library.Library.GetTrackTable')).toEqual([8]); expect(lastArgs('library.Library.GetTracks')).toEqual([8]);
}); });
it('picks up a library added while it was on screen', async () => { it('picks up a library added while it was on screen', async () => {
@@ -72,7 +72,7 @@ function boxOf(el: Element | null | undefined): { w: number; h: number } {
describe('the way out of a detail view', () => { describe('the way out of a detail view', () => {
beforeEach(() => { beforeEach(() => {
for (const path of [ for (const path of [
'library.Library.GetTrackTable', 'library.Library.GetTracks',
'library.Library.GetAlbums', 'library.Library.GetAlbums',
'library.Library.GetArtists', 'library.Library.GetArtists',
'library.Library.GetGenres', 'library.Library.GetGenres',
@@ -11,12 +11,11 @@ import '@components/track-list/track-list';
import { Events } from '../../src/events'; import { Events } from '../../src/events';
import { emit, stub, stubFailure, flush, resetHarness } from '@test/support/harness'; import { emit, stub, stubFailure, flush, resetHarness } from '@test/support/harness';
import { fixture, shadow, text } from '@test/support/render'; import { fixture, shadow, text } from '@test/support/render';
import { trackTable } from '@test/support/track-table';
/** Drop the library store's cache so the list has to fetch. */ /** Drop the library store's cache so the list has to fetch. */
async function emptyLibrary(): Promise<void> { async function emptyLibrary(): Promise<void> {
resetHarness(); resetHarness();
stub('library.Library.GetTrackTable', trackTable([])); stub('library.Library.GetTracks', []);
stub('library.Library.GetAlbums', []); stub('library.Library.GetAlbums', []);
stub('library.Library.GetArtists', []); stub('library.Library.GetArtists', []);
stub('library.Library.GetGenres', []); stub('library.Library.GetGenres', []);
@@ -41,7 +40,7 @@ describe('<track-list> empty, loading and failed', () => {
}); });
it('says the query failed, and offers to try again', async () => { it('says the query failed, and offers to try again', async () => {
stubFailure('library.Library.GetTrackTable', 'sql: database is locked'); stubFailure('library.Library.GetTracks', 'sql: database is locked');
emit(Events.LibraryScanComplete); emit(Events.LibraryScanComplete);
await flush(); await flush();
@@ -28,7 +28,7 @@ import type { TrackDetails } from '@components/track-details/track-details';
const ALBUM_TRACKS = 'library.Library.GetAlbumTracks'; const ALBUM_TRACKS = 'library.Library.GetAlbumTracks';
const COMPLETENESS = 'library.Library.GetAlbumCompleteness'; const COMPLETENESS = 'library.Library.GetAlbumCompleteness';
const FILE_PATHS = 'library.Library.GetFilePathsByRecordingMBIDs'; const FILE_PATHS = 'library.Library.GetFilePathsByRecordingMBIDs';
const TRACKS_BY_PATH = 'library.Library.GetTracksByPaths'; const ALL_TRACKS = 'library.Library.GetTracks';
const LOOKUP_RG = 'explore.Service.LookupReleaseGroup'; const LOOKUP_RG = 'explore.Service.LookupReleaseGroup';
const BROWSE_RELEASES = 'explore.Service.BrowseReleases'; const BROWSE_RELEASES = 'explore.Service.BrowseReleases';
@@ -127,13 +127,14 @@ describe('Explore track details', () => {
stub(ALBUM_TRACKS, [albumTrack]); stub(ALBUM_TRACKS, [albumTrack]);
stub(COMPLETENESS, { known: true, complete: true, owned: 1, expected: 1 }); stub(COMPLETENESS, { known: true, complete: true, owned: 1, expected: 1 });
stub(FILE_PATHS, { [MBID]: [PATH] }); stub(FILE_PATHS, { [MBID]: [PATH] });
stub(TRACKS_BY_PATH, [libraryTrack]); stub(ALL_TRACKS, [libraryTrack]);
}); });
/** /**
* `trackCache` keeps what it was answered for the life of the browser * `libraryStore` fetches at import and caches the empty list the
* session, so a test that changes the answer has to drop it. A * shared setup stubs, for the life of the browser session — so a test
* scan-complete event is how the app itself invalidates that cache. * that wants tracks in it has to say so. A scan-complete event is how
* the app itself invalidates that cache.
*/ */
async function primeLibrary() { async function primeLibrary() {
emit(Events.LibraryScanComplete); emit(Events.LibraryScanComplete);
@@ -203,7 +204,7 @@ describe('Explore track details', () => {
items[details]!.dispatchEvent(new MouseEvent('click', { bubbles: true })); items[details]!.dispatchEvent(new MouseEvent('click', { bubbles: true }));
// Polled rather than counted: the opener is three awaits deep — the // Polled rather than counted: the opener is three awaits deep — the
// path lookup, the track by path, and the dynamic `import()` of // path lookup, the store's tracks, and the dynamic `import()` of
// the dialog chunk — and a chunk fetch is the one of the three // the dialog chunk — and a chunk fetch is the one of the three
// whose cost depends on what else the suite is doing. // whose cost depends on what else the suite is doing.
const shown = await dialogTrack(el); const shown = await dialogTrack(el);
@@ -229,7 +230,7 @@ describe('Explore track details', () => {
}); });
it('reports a path with no library track rather than opening empty', async () => { it('reports a path with no library track rather than opening empty', async () => {
stub(TRACKS_BY_PATH, []); stub(ALL_TRACKS, []);
await primeLibrary(); await primeLibrary();
const outcome = await showTrackDetailsForPath( const outcome = await showTrackDetailsForPath(
@@ -14,7 +14,6 @@ import '@components/track-list/track-list';
import { stub, emit, flush } from '@test/support/harness'; import { stub, emit, flush } from '@test/support/harness';
import { Events } from '../../src/events'; import { Events } from '../../src/events';
import { fixture, shadow, shadowAll, update } from '@test/support/render'; import { fixture, shadow, shadowAll, update } from '@test/support/render';
import { trackTable } from '@test/support/track-table';
/** Two fixture tracks, enough to move a focus ring between. */ /** Two fixture tracks, enough to move a focus ring between. */
const TRACKS = [ const TRACKS = [
@@ -79,7 +78,7 @@ describe('<queue-panel> when closed', () => {
describe('<track-list> roving tabindex', () => { describe('<track-list> roving tabindex', () => {
beforeEach(() => { beforeEach(() => {
stub('library.Library.GetTrackTable', trackTable(TRACKS)); stub('library.Library.GetTracks', TRACKS);
}); });
it('offers exactly one tab stop, however many rows there are', async () => { it('offers exactly one tab stop, however many rows there are', async () => {
@@ -51,12 +51,8 @@ describe('track-details stays out of the startup chunk', () => {
}, },
); );
// Directly, or through `utils/track-details-opener`, which awaits
// it and is what the path-keyed openers share (#279).
it.each(OPENERS)('%s loads it at the point of use', (_name, source) => { it.each(OPENERS)('%s loads it at the point of use', (_name, source) => {
expect(source).toMatch( expect(source).toContain('loadTrackDetails');
/loadTrackDetails|showTrackDetailsForPath|showBatchTrackDetailsForPaths/,
);
}); });
}); });
@@ -31,7 +31,6 @@ import {
stubFailure, stubFailure,
} from '@test/support/harness'; } from '@test/support/harness';
import { fixture, shadow, shadowAll, update } from '@test/support/render'; import { fixture, shadow, shadowAll, update } from '@test/support/render';
import { trackTable } from '@test/support/track-table';
const SEARCH = 'explore.Service.SearchLocal'; const SEARCH = 'explore.Service.SearchLocal';
@@ -134,7 +133,7 @@ describe('<explore-view> badges', () => {
stub('explore.Service.GetArtistImageURL', ''); stub('explore.Service.GetArtistImageURL', '');
stub('explore.Service.GetExploreShelves', { shelves: [], state: 'ready' }); stub('explore.Service.GetExploreShelves', { shelves: [], state: 'ready' });
stub('library.Library.GetAlbums', []); stub('library.Library.GetAlbums', []);
stub('library.Library.GetTrackTable', trackTable([])); stub('library.Library.GetTracks', []);
await withRequests([]); await withRequests([]);
}); });
@@ -48,12 +48,21 @@ describe('the track list Art column', () => {
]).toEqual(['lazy', 'async']); ]).toEqual(['lazy', 'async']);
}); });
// There is no fallback through the larger tiers any more: the list it('falls back through the tiers rather than rendering nothing', () => {
// carries only the small one (#281), and the backend derives every const onlyOriginal = cell('albumArt', {
// tier from the same file, so it is set whenever any of them is. CoverArtPath: '/covers/abc.jpg',
CoverArtSmall: '',
CoverArtMedium: '',
}).querySelector('img');
expect(onlyOriginal?.getAttribute('src')).toBe('/covers/abc.jpg');
});
it('renders nothing at all when there is no art', () => { it('renders nothing at all when there is no art', () => {
const none = cell('albumArt', { const none = cell('albumArt', {
CoverArtPath: '',
CoverArtSmall: '', CoverArtSmall: '',
CoverArtMedium: '',
}); });
expect(none.querySelector('img')).toBeNull(); expect(none.querySelector('img')).toBeNull();
@@ -1,86 +0,0 @@
/**
* The rule editor's value box suggests values from the library by
* asking for the ones that match what has been typed (#279).
*
* It used to build the lists from `libraryStore`'s whole-library
* arrays: one more reason to load every track at startup, and an empty
* list whenever they had not loaded. These tests stub *no* library
* collection — what is asserted is that the suggestions arrive anyway,
* from the one call made for them.
*/
import { describe, expect, it, beforeEach } from 'vitest';
import '@components/smart-playlist-editor/smart-playlist-editor';
import type { YjCombobox } from '@components/combobox/combobox';
import { calls, resetHarness, stub } from '@test/support/harness';
import { fixture, shadow, shadowAll } from '@test/support/render';
import type { LitElement } from 'lit';
const SUGGEST = 'playlist.Service.SuggestSmartPlaylistValues';
async function editor(field: string): Promise<LitElement> {
return fixture<LitElement>('smart-playlist-editor', {
rules: JSON.stringify({ rules: [{ field, operator: 'contains', value: '' }] }),
});
}
/** The row's second combobox: the value box (the first picks the field). */
function valueBox(el: LitElement): YjCombobox {
return shadowAll<YjCombobox>(el, 'yj-combobox')[1]!;
}
async function type(box: YjCombobox, text: string): Promise<void> {
const input = shadow<HTMLInputElement>(box, 'input')!;
input.focus();
input.value = text;
input.dispatchEvent(new Event('input', { bubbles: true }));
await box.updateComplete;
}
describe('smart playlist value suggestions', () => {
beforeEach(() => {
resetHarness();
stub(SUGGEST, (field: string, needle: string) =>
[`${field}:${needle}:1`, `${field}:${needle}:2`],
);
});
it('asks for the values matching what was typed, once typing pauses', async () => {
const el = await editor('artist');
const box = valueBox(el);
await type(box, 'q');
await type(box, 'qu');
await expect.poll(() => box.options).toEqual(['artist:qu:1', 'artist:qu:2']);
// Debounced: the pause after "qu" asked, the keystroke before did not.
expect(calls(SUGGEST).map((c) => c.args)).toEqual([['artist', 'qu']]);
});
it('answers a repeat from what it already fetched', async () => {
const el = await editor('album');
const box = valueBox(el);
await type(box, 'x');
await expect.poll(() => box.options).toEqual(['album:x:1', 'album:x:2']);
await type(box, 'xy');
await expect.poll(() => box.options).toEqual(['album:xy:1', 'album:xy:2']);
await type(box, 'x');
expect(box.options).toEqual(['album:x:1', 'album:x:2']);
expect(calls(SUGGEST)).toHaveLength(2);
});
it('asks nothing for a field that has no suggestions', async () => {
const el = await editor('duration');
const boxes = shadowAll<YjCombobox>(el, 'yj-combobox');
// A numeric range field may render no value combobox at all; if it
// does, typing in it must not ask.
if (boxes[1]) await type(boxes[1], '3');
await new Promise((r) => setTimeout(r, 250));
expect(calls(SUGGEST)).toHaveLength(0);
});
});
+1 -1
View File
@@ -117,7 +117,7 @@ const TAGS = [
*/ */
function stubEmptyBackend(): void { function stubEmptyBackend(): void {
const emptyLists = [ const emptyLists = [
'library.Library.GetTrackTable', 'library.Library.GetTracks',
'library.Library.GetAlbums', 'library.Library.GetAlbums',
'library.Library.GetArtists', 'library.Library.GetArtists',
'library.Library.GetGenres', 'library.Library.GetGenres',
@@ -24,7 +24,6 @@ import '@components/selection-bar/selection-bar';
import { calls, flush, resetHarness, stub } from '@test/support/harness'; import { calls, flush, resetHarness, stub } from '@test/support/harness';
import { fixture, shadow, shadowAll } from '@test/support/render'; import { fixture, shadow, shadowAll } from '@test/support/render';
import { installTouchGestures, LONG_PRESS_MS } from '@utils/touch-gestures'; import { installTouchGestures, LONG_PRESS_MS } from '@utils/touch-gestures';
import { trackTable } from '@test/support/track-table';
const HELD = LONG_PRESS_MS + 120; const HELD = LONG_PRESS_MS + 120;
@@ -118,7 +117,7 @@ function rows(el: HTMLElement): HTMLElement[] {
describe('a finger on a track row', () => { describe('a finger on a track row', () => {
beforeEach(() => { beforeEach(() => {
resetHarness(); resetHarness();
stub('library.Library.GetTrackTable', trackTable(TRACKS)); stub('library.Library.GetTracks', TRACKS);
stub('library.Library.GetAllLibrariesWithTrackCounts', []); stub('library.Library.GetAllLibrariesWithTrackCounts', []);
stub('config.Config.GetShortcuts', {}); stub('config.Config.GetShortcuts', {});
stub('queue.Queue.SetQueue', null); stub('queue.Queue.SetQueue', null);
@@ -321,7 +320,7 @@ describe('<selection-bar>', () => {
describe('a tap on a name inside a row', () => { describe('a tap on a name inside a row', () => {
beforeEach(() => { beforeEach(() => {
resetHarness(); resetHarness();
stub('library.Library.GetTrackTable', trackTable(TRACKS)); stub('library.Library.GetTracks', TRACKS);
stub('library.Library.GetAllLibrariesWithTrackCounts', []); stub('library.Library.GetAllLibrariesWithTrackCounts', []);
stub('config.Config.GetShortcuts', {}); stub('config.Config.GetShortcuts', {});
stub('queue.Queue.SetQueue', null); stub('queue.Queue.SetQueue', null);
+1 -2
View File
@@ -33,7 +33,6 @@ import '@components/track-list/track-list';
import { calls, flush, resetHarness, stub } from '@test/support/harness'; import { calls, flush, resetHarness, stub } from '@test/support/harness';
import { fixture, shadow, shadowAll } from '@test/support/render'; import { fixture, shadow, shadowAll } from '@test/support/render';
import { installTouchGestures } from '@utils/touch-gestures'; import { installTouchGestures } from '@utils/touch-gestures';
import { trackTable } from '@test/support/track-table';
const wait = (ms: number) => new Promise((r) => setTimeout(r, ms)); const wait = (ms: number) => new Promise((r) => setTimeout(r, ms));
@@ -126,7 +125,7 @@ function threshold(row: HTMLElement): number {
describe('a finger swiped right across a track row', () => { describe('a finger swiped right across a track row', () => {
beforeEach(() => { beforeEach(() => {
resetHarness(); resetHarness();
stub('library.Library.GetTrackTable', trackTable(TRACKS)); stub('library.Library.GetTracks', TRACKS);
stub('library.Library.GetAllLibrariesWithTrackCounts', []); stub('library.Library.GetAllLibrariesWithTrackCounts', []);
stub('config.Config.GetShortcuts', {}); stub('config.Config.GetShortcuts', {});
stub('queue.Queue.SetQueue', null); stub('queue.Queue.SetQueue', null);
@@ -231,7 +231,7 @@ const CACHED_VIEWS = [
* binding resolves undefined, which is not what Go sends. */ * binding resolves undefined, which is not what Go sends. */
function stubEmptyBackend(): void { function stubEmptyBackend(): void {
for (const path of [ for (const path of [
'library.Library.GetTrackTable', 'library.Library.GetTracks',
'library.Library.GetAlbums', 'library.Library.GetAlbums',
'library.Library.GetArtists', 'library.Library.GetArtists',
'library.Library.GetGenres', 'library.Library.GetGenres',
+1 -1
View File
@@ -35,7 +35,7 @@ const importTimeDefaults: Array<[string, unknown]> = [
// libraryStore and playlistStore fetch eagerly at import. Left // libraryStore and playlistStore fetch eagerly at import. Left
// unstubbed they would cache `undefined` — not the empty list Go // unstubbed they would cache `undefined` — not the empty list Go
// sends — and every consumer would then crash on `.length`. // sends — and every consumer would then crash on `.length`.
['library.Library.GetTrackTable', { strings: [''] }], ['library.Library.GetTracks', []],
['library.Library.GetAlbums', []], ['library.Library.GetAlbums', []],
['library.Library.GetArtists', []], ['library.Library.GetArtists', []],
['library.Library.GetGenres', []], ['library.Library.GetGenres', []],
-99
View File
@@ -1,99 +0,0 @@
/**
* #280: the app loads the library when a view needs it, not at startup.
*
* Every collection used to be fetched together at `DOMContentLoaded`
* and refetched on every invalidation, whether or not anything was
* showing it. On a 26 138-track library the track list was 20.5 MB of
* that and cost the backend ~170 MB of transient allocation — for
* someone looking at Home, which draws none of it.
*
* **This file must not load the track list before the first test.** It
* asserts the state the app is actually left in by its own startup, so
* a test added above that asks for tracks would invalidate the
* precondition rather than silently pass.
*/
import { describe, expect, it, beforeEach } from 'vitest';
import { libraryStore } from '@store/library-store';
import { Events } from '../../src/events';
import {
calls,
emit,
flush,
resetHarness,
stub,
} from '@test/support/harness';
const TRACKS = 'library.Library.GetTrackTable';
const SMALL = [
'library.Library.GetAlbums',
'library.Library.GetArtists',
'library.Library.GetGenres',
];
/**
* Wait for the store's own idle warm-up to land.
*
* It asks for the three small collections on `requestIdleCallback`, so
* when it has happened is the browser's decision — waiting for the
* effect rather than for a duration is the only way this is not a race.
*/
function stubReads(): void {
stub(TRACKS, { strings: [''] });
for (const path of SMALL) stub(path, []);
}
async function waitForWarm(): Promise<void> {
for (let i = 0; i < 200 && libraryStore.getCachedAlbums() === null; i++) {
await flush();
}
expect(libraryStore.getCachedAlbums(), 'the warm-up ran').not.toBeNull();
}
describe('what the app loads on its own (#280)', () => {
beforeEach(async () => {
stubReads();
await waitForWarm();
// resetHarness clears the stubs as well as the recorded calls.
resetHarness();
stubReads();
});
it('leaves the track list alone', () => {
expect(libraryStore.getCachedTracks()).toBeNull();
expect(calls(TRACKS)).toEqual([]);
});
it('does not load it to answer a scan', async () => {
emit(Events.LibraryScanComplete);
await flush();
// The small collections are in use, so they are refreshed; the
// track list nobody has opened is not.
expect(calls().map((c) => c.path).sort()).toEqual([...SMALL].sort());
});
it('loads it for the view that draws it, and refreshes it from then on', async () => {
libraryStore.prefetch('tracks');
await flush();
expect(calls(TRACKS)).toHaveLength(1);
expect(libraryStore.getCachedTracks()).not.toBeNull();
resetHarness();
emit(Events.LibraryScanComplete);
await flush();
expect(calls(TRACKS), 'in use now, so a scan refreshes it').toHaveLength(1);
});
it('is asked for nothing by a prefetch for a view it has no data for', async () => {
libraryStore.prefetch('home');
libraryStore.prefetch('settings');
libraryStore.prefetch('explore');
await flush();
expect(calls()).toEqual([]);
});
});
+27 -124
View File
@@ -19,17 +19,10 @@ import {
lastArgs, lastArgs,
resetHarness, resetHarness,
} from '@test/support/harness'; } from '@test/support/harness';
import { trackTable } from '@test/support/track-table';
const RETAGGED = {
TrackName: 'Retagged',
FilePath: '/a.mp3',
PlayCount: 0,
};
const TRACKS = [ const TRACKS = [
{ TrackName: 'One', FilePath: '/a.mp3', PlayCount: 0 }, { ID: 1, Title: 'One', FilePath: '/a.mp3', PlayCount: 0, LastPlayed: '' },
{ TrackName: 'Two', FilePath: '/b.mp3', PlayCount: 4 }, { ID: 2, Title: 'Two', FilePath: '/b.mp3', PlayCount: 4, LastPlayed: 'x' },
]; ];
const ALBUMS = [{ ID: 1, Name: 'Album', ArtistName: 'Artist' }]; const ALBUMS = [{ ID: 1, Name: 'Album', ArtistName: 'Artist' }];
const OTHER_ALBUMS = [{ ID: 2, Name: 'Other', ArtistName: 'Other Artist' }]; const OTHER_ALBUMS = [{ ID: 2, Name: 'Other', ArtistName: 'Other Artist' }];
@@ -40,34 +33,30 @@ const LIBRARIES = [{ id: 7, name: 'Music' }, { id: 8, name: 'Field' }];
/** Stub every read binding the store can reach. Unstubbed bindings /** Stub every read binding the store can reach. Unstubbed bindings
* resolve undefined, which the store would cache as if it were data. */ * resolve undefined, which the store would cache as if it were data. */
function stubReads(): void { function stubReads(): void {
stub('library.Library.GetTrackTable', trackTable(TRACKS)); stub('library.Library.GetTracks', TRACKS);
stub('library.Library.GetAlbums', ALBUMS); stub('library.Library.GetAlbums', ALBUMS);
stub('library.Library.GetArtists', ARTISTS); stub('library.Library.GetArtists', ARTISTS);
stub('library.Library.GetGenres', GENRES); stub('library.Library.GetGenres', GENRES);
stub('library.Library.GetTracks', TRACKS);
stub('library.Library.GetAlbums', ALBUMS);
stub('library.Library.GetArtists', ARTISTS);
stub('library.Library.GetGenres', GENRES);
stub('library.Library.GetAlbumsByArtist', ALBUMS);
stub('library.Library.GetAlbumsByArtist', ALBUMS); stub('library.Library.GetAlbumsByArtist', ALBUMS);
stub('library.Library.GetAllLibrariesWithTrackCounts', LIBRARIES); stub('library.Library.GetAllLibrariesWithTrackCounts', LIBRARIES);
} }
/** /**
* Start each test from a loaded store. The store has no reset of its * Drop the cache and let the eager refetch settle, so each test starts
* own; a scan completing is how the app itself clears it — and since * from the same place. The store has no reset of its own; a scan
* #280 it reloads only what something had loaded, so the four reads * completing is how the app itself clears it.
* here are what says "this test is about a loaded store".
*/ */
async function reload(): Promise<void> { async function reload(): Promise<void> {
stubReads(); stubReads();
emit(Events.LibraryScanComplete); emit(Events.LibraryScanComplete);
await flush(); await flush();
resetHarness(); // The eager refetch the invalidation kicks off is recorded like any
stubReads(); // other call; clear it, or every count in every test is off by one.
await Promise.all([
libraryStore.getTracks(),
libraryStore.getAlbums(),
libraryStore.getArtists(),
libraryStore.getGenres(),
]);
// Those reads are recorded like any other call; clear them, or every
// count in every test is off by one.
resetHarness(); resetHarness();
stubReads(); stubReads();
} }
@@ -80,7 +69,7 @@ describe('library store: caching', () => {
it('serves a second read from cache without touching the backend', async () => { it('serves a second read from cache without touching the backend', async () => {
await libraryStore.getTracks(); await libraryStore.getTracks();
expect(calls('library.Library.GetTrackTable')).toHaveLength(0); expect(calls('library.Library.GetTracks')).toHaveLength(0);
}); });
it('deduplicates concurrent first reads into one backend call', async () => { it('deduplicates concurrent first reads into one backend call', async () => {
@@ -99,14 +88,14 @@ describe('library store: caching', () => {
it('exposes cached collections synchronously once loaded', () => { it('exposes cached collections synchronously once loaded', () => {
expect([ expect([
libraryStore.getCachedTracks()?.map((t) => t.FilePath), libraryStore.getCachedTracks(),
libraryStore.getCachedAlbums(), libraryStore.getCachedAlbums(),
libraryStore.cachedArtists, libraryStore.cachedArtists,
libraryStore.getCachedGenres(), libraryStore.getCachedGenres(),
]).toEqual([['/a.mp3', '/b.mp3'], ALBUMS, ARTISTS, GENRES]); ]).toEqual([TRACKS, ALBUMS, ARTISTS, GENRES]);
}); });
it('refetches everything a view had loaded when a scan completes', async () => { it('refetches everything when a scan completes', async () => {
emit(Events.LibraryScanComplete); emit(Events.LibraryScanComplete);
await flush(); await flush();
@@ -114,98 +103,15 @@ describe('library store: caching', () => {
'library.Library.GetAlbums', 'library.Library.GetAlbums',
'library.Library.GetArtists', 'library.Library.GetArtists',
'library.Library.GetGenres', 'library.Library.GetGenres',
'library.Library.GetTrackTable', 'library.Library.GetTracks',
]); ]);
}); });
/* it('refetches when a track is retagged', async () => {
* #282: a retag names the one file it rewrote, so the whole list does
* not have to come back — 20.5 MB at 26 138 tracks to pick up one new
* title. The summaries are still refetched, because a retag can move
* a track between albums.
*/
it('re-reads the one file a retag names, not the whole list', async () => {
stub('library.Library.GetTracksByPaths', [RETAGGED]);
emit(Events.TrackMetadataChanged, { filePath: '/a.mp3' }); emit(Events.TrackMetadataChanged, { filePath: '/a.mp3' });
await flush(); await flush();
expect(calls('library.Library.GetTrackTable')).toHaveLength(0); expect(calls('library.Library.GetTracks')).toHaveLength(1);
expect(lastArgs('library.Library.GetTracksByPaths')).toEqual([['/a.mp3']]);
expect(calls().map((c) => c.path).sort()).toEqual([
'library.Library.GetAlbums',
'library.Library.GetArtists',
'library.Library.GetGenres',
'library.Library.GetTracksByPaths',
]);
});
it('splices the re-read row in, so consumers notice', async () => {
stub('library.Library.GetTracksByPaths', [RETAGGED]);
const before = libraryStore.getCachedTracks();
emit(Events.TrackMetadataChanged, { filePath: '/a.mp3' });
await flush();
const after = libraryStore.getCachedTracks();
expect(after?.map((t) => t.TrackName)).toEqual(['Retagged', 'Two']);
expect(after, 'a new array, which is what memoized consumers key on').not.toBe(
before,
);
});
it('falls back to a full reload when the patch cannot be read', async () => {
stubFailure('library.Library.GetTracksByPaths', 'sql: database is locked');
emit(Events.TrackMetadataChanged, { filePath: '/a.mp3' });
await flush();
expect(calls('library.Library.GetTrackTable')).toHaveLength(1);
});
it('reloads everything for a batch write, which names no paths', async () => {
emit(Events.TrackMetadataChanged, { batch: true, total: 40 });
await flush();
expect(calls('library.Library.GetTrackTable')).toHaveLength(1);
expect(calls('library.Library.GetTracksByPaths')).toHaveLength(0);
});
/*
* #282, the other half: a soft rescan of an unchanged library reports
* zero added, updated and removed, and every loaded collection was
* thrown away and refetched anyway.
*/
describe('a scan that changed nothing', () => {
beforeEach(async () => {
emit(Events.LibraryScanComplete, {
added: 0,
updated: 0,
removed: 0,
skipped: 31,
});
await flush();
});
it('refetches nothing', () => {
expect(calls()).toEqual([]);
});
it('keeps the data it already had', () => {
expect(libraryStore.getCachedTracks()?.map((t) => t.FilePath)).toEqual([
'/a.mp3',
'/b.mp3',
]);
});
});
it('reloads when a scan reports a change', async () => {
emit(Events.LibraryScanComplete, { added: 1, updated: 0, removed: 0 });
await flush();
expect(calls('library.Library.GetTrackTable')).toHaveLength(1);
}); });
/* /*
@@ -234,11 +140,10 @@ describe('library store: caching', () => {
it('patches the one track it names', () => { it('patches the one track it names', () => {
const tracks = libraryStore.getCachedTracks(); const tracks = libraryStore.getCachedTracks();
// LastPlayed is not on the list's rows (#281); trackCache
// patches it on whole tracks.
expect(tracks?.[0]).toMatchObject({ expect(tracks?.[0]).toMatchObject({
FilePath: '/a.mp3', FilePath: '/a.mp3',
PlayCount: 9, PlayCount: 9,
LastPlayed: '2026-08-11 10:00:00',
}); });
}); });
@@ -287,7 +192,7 @@ describe('library store: caching', () => {
}); });
it('does not refetch the tracks', () => { it('does not refetch the tracks', () => {
expect(calls('library.Library.GetTrackTable')).toHaveLength(0); expect(calls('library.Library.GetTracks')).toHaveLength(0);
}); });
it('splices the removed track out in place', () => { it('splices the removed track out in place', () => {
@@ -353,7 +258,7 @@ describe('library store: library filter', () => {
libraryStore.setSelectedLibrary(7); libraryStore.setSelectedLibrary(7);
await flush(); await flush();
expect(lastArgs('library.Library.GetTrackTable')).toEqual([7]); expect(lastArgs('library.Library.GetTracks')).toEqual([7]);
}); });
it('ignores a redundant selection instead of invalidating', async () => { it('ignores a redundant selection instead of invalidating', async () => {
@@ -399,12 +304,12 @@ describe('library store: a fetch that is overtaken', () => {
it('serves the library that is selected, not the one that was in flight', async () => { it('serves the library that is selected, not the one that was in flight', async () => {
const pending: Array<{ id: number; resolve: (v: unknown) => void }> = []; const pending: Array<{ id: number; resolve: (v: unknown) => void }> = [];
const byLibrary = (id: number) => trackTable([{ FilePath: `/lib-${id}.mp3` }]); const byLibrary = (id: number) => [{ ID: id, Title: `Library ${id}` }];
// Only the track fetch is held open; the other three settle at once, // Only the track fetch is held open; the other three settle at once,
// so the test is about the overtaking and nothing else. // so the test is about the overtaking and nothing else.
stub( stub(
'library.Library.GetTrackTable', 'library.Library.GetTracks',
(id: number) => (id: number) =>
new Promise((resolve) => { new Promise((resolve) => {
pending.push({ id, resolve }); pending.push({ id, resolve });
@@ -422,13 +327,11 @@ describe('library store: a fetch that is overtaken', () => {
pending.find((p) => p.id === 8)?.resolve(byLibrary(8)); pending.find((p) => p.id === 8)?.resolve(byLibrary(8));
await flush(); await flush();
expect(libraryStore.getCachedTracks()?.map((t) => t.FilePath)).toEqual([ expect(libraryStore.getCachedTracks()).toEqual(byLibrary(8));
'/lib-8.mp3',
]);
}); });
it('settles the waiters when the fetch they are waiting on fails', async () => { it('settles the waiters when the fetch they are waiting on fails', async () => {
stubFailure('library.Library.GetTrackTable', 'sql: database is locked'); stubFailure('library.Library.GetTracks', 'sql: database is locked');
// Invalidation drops the cache and starts the fetch that fails. // Invalidation drops the cache and starts the fetch that fails.
emit(Events.LibraryScanComplete); emit(Events.LibraryScanComplete);
-209
View File
@@ -1,209 +0,0 @@
/**
* `trackCache` answers "which track is at this path" for the rows a
* surface is showing (#279), instead of every surface reaching into the
* whole library's array — which had to be fetched eagerly at startup
* for that to work, and silently did nothing when it had not been.
*/
import { describe, expect, it, beforeEach } from 'vitest';
import { trackCache } from '@store/track-cache';
import { libraryStore } from '@store/library-store';
import { showBatchTrackDetailsForPaths } from '@utils/track-details-opener';
import type { TrackDetails } from '@components/track-details/track-details';
import { Events } from '../../src/events';
import {
calls,
emit,
flush,
resetHarness,
stub,
stubFailure,
} from '@test/support/harness';
const BY_PATHS = 'library.Library.GetTracksByPaths';
function track(path: string, album = 'Album') {
return {
FilePath: path,
TrackName: path,
ArtistName: 'Artist',
Album: album,
PlayCount: 0,
LastPlayed: '',
CoverArtPath: '',
};
}
/** The backend: answers every path that starts with /lib/. */
function stubLibrary(): void {
stub(BY_PATHS, (paths: string[]) =>
paths.filter((p) => p.startsWith('/lib/')).map((p) => track(p)),
);
}
describe('track cache', () => {
beforeEach(() => {
// The cache is a singleton; a scan completing is how the app empties it.
emit(Events.LibraryScanComplete);
resetHarness();
stubLibrary();
});
it('answers in the order asked and leaves out paths not in the library', async () => {
const got = await trackCache.get(['/lib/b', '/gone', '/lib/a']);
expect(got.map((t) => t.FilePath)).toEqual(['/lib/b', '/lib/a']);
});
it('coalesces lookups made together into one call', async () => {
await Promise.all([
trackCache.get(['/lib/a']),
trackCache.get(['/lib/b', '/lib/a']),
trackCache.getOne('/lib/c'),
]);
expect(calls(BY_PATHS)).toHaveLength(1);
expect((calls(BY_PATHS)[0]!.args[0] as string[]).sort()).toEqual([
'/lib/a',
'/lib/b',
'/lib/c',
]);
});
it('answers a repeat from cache, and asks again after a retag of that file', async () => {
await trackCache.get(['/lib/a', '/lib/b']);
await trackCache.get(['/lib/a', '/lib/b']);
expect(calls(BY_PATHS)).toHaveLength(1);
emit(Events.TrackMetadataChanged, { filePath: '/lib/a' });
await trackCache.get(['/lib/b']);
expect(calls(BY_PATHS), 'an unchanged file is still cached').toHaveLength(1);
await trackCache.get(['/lib/a']);
expect(calls(BY_PATHS)).toHaveLength(2);
});
it('patches a play count in place without a call', async () => {
await trackCache.getOne('/lib/a');
emit(Events.TrackPlayCountChanged, {
filePath: '/lib/a',
playCount: 7,
lastPlayed: '2026-10-05 10:00:00',
});
const got = await trackCache.getOne('/lib/a');
expect([got?.PlayCount, got?.LastPlayed, calls(BY_PATHS).length]).toEqual([
7,
'2026-10-05 10:00:00',
1,
]);
});
it('does not cache an answer that crossed an invalidation', async () => {
let answer: (v: unknown) => void = () => undefined;
stub(BY_PATHS, () => new Promise((resolve) => {
answer = resolve;
}));
const pending = trackCache.getOne('/lib/a');
// Sent, not yet answered: the invalidation lands in between.
await flush();
expect(calls(BY_PATHS)).toHaveLength(1);
emit(Events.TrackMetadataChanged, { batch: true });
answer([track('/lib/a')]);
expect((await pending)?.FilePath, 'the caller still gets it').toBe('/lib/a');
stubLibrary();
await trackCache.getOne('/lib/a');
expect(calls(BY_PATHS)).toHaveLength(2);
});
it('rejects every waiter in a batch when the call fails', async () => {
stubFailure(BY_PATHS);
const results = await Promise.allSettled([
trackCache.getOne('/lib/a'),
trackCache.getOne('/lib/b'),
]);
expect(results.map((r) => r.status)).toEqual(['rejected', 'rejected']);
});
});
/*
* The bug half of #279: the queue, playlist and smart-playlist openers
* read `libraryStore.getCachedTracks()` and returned silently when it
* was null. The batch opener is what all three call now, so it is
* asserted against a library store that has nothing loaded.
*/
describe('opening details with no library list loaded', () => {
beforeEach(() => {
emit(Events.LibraryScanComplete);
resetHarness();
stubLibrary();
stub('library.Library.GetTracks', []);
});
it('shows the dialog with the tracks asked for', async () => {
await flush();
expect(libraryStore.getCachedTracks()?.length ?? 0).toBe(0);
let shown: unknown[] | null = null;
const dialog = {
showBatch: (tracks: unknown[]) => {
shown = tracks;
},
} as unknown as TrackDetails;
const outcome = await showBatchTrackDetailsForPaths(
() => dialog,
['/lib/a', '/lib/b'],
() => undefined,
);
expect(outcome).toBe('shown');
expect((shown ?? []).map((t) => (t as { FilePath: string }).FilePath)).toEqual([
'/lib/a',
'/lib/b',
]);
});
});
/** Every source file, as text. */
const SOURCES = import.meta.glob<string>('../../src/**/*.ts', {
eager: true,
query: '?raw',
import: 'default',
});
/**
* Who may read the whole library's track array. The store owns it and
* the Tracks view draws it; anything else that needs a track by path
* asks `trackCache`, or the array becomes a startup dependency again.
*/
const MAY_READ_ALL_TRACKS = [
'store/library-store.ts',
'store/controllers/library-controller.ts',
'components/track-list/track-list.ts',
];
const ALL_TRACKS_READ = /getCachedTracks\(|libraryStore\.getTracks\(|\bcachedTracks\b/;
describe('the whole-library track array has one reader', () => {
it('reads the sources it sweeps', () => {
expect(Object.keys(SOURCES).length).toBeGreaterThan(100);
});
it('is read only by the store and the Tracks view', () => {
const readers = Object.entries(SOURCES)
.filter(([, src]) => ALL_TRACKS_READ.test(src))
.map(([path]) => path.replace('../../src/', ''))
.sort();
expect(readers).toEqual([...MAY_READ_ALL_TRACKS].sort());
});
});
-86
View File
@@ -1,86 +0,0 @@
/**
* Encode test tracks as the backend's `TrackTable` (#281), so a stub of
* `library.Library.GetTrackTable` can be written as a list of tracks.
*
* Test fixtures are partial tracks; a missing field encodes as the
* zero value Go would have sent. The encoding mirrors
* `backend/library/tracktable.go` closely enough to exercise the real
* decoder: shared string table with "" at 0, genre lists interned.
*/
import type * as library from '@go/library/models.js';
/** A fixture: any subset of a track's fields, loosely typed as fixtures are. */
type PartialTrack = { FilePath: string } & Record<string, unknown>;
const STRING_COLS = [
['trackName', 'TrackName'],
['artistName', 'ArtistName'],
['album', 'Album'],
['composer', 'Composer'],
['fileType', 'FileType'],
['artistMbid', 'ArtistMBID'],
['releaseGroupMbid', 'ReleaseGroupMBID'],
['recordingMbid', 'RecordingMBID'],
['coverArtSmall', 'CoverArtSmall'],
] as const;
const INT_COLS = [
['trackNumber', 'TrackNumber'],
['discNumber', 'DiscNumber'],
['year', 'Year'],
['sampleRate', 'SampleRate'],
['bitDepth', 'BitDepth'],
['channels', 'Channels'],
['bitrate', 'Bitrate'],
['fileSize', 'FileSize'],
['playCount', 'PlayCount'],
] as const;
export function trackTable(tracks: readonly PartialTrack[]): library.TrackTable {
const strings = [''];
const index = new Map<string, number>([['', 0]]);
const intern = (s: string | undefined): number => {
const v = s ?? '';
let i = index.get(v);
if (i === undefined) {
i = strings.length;
strings.push(v);
index.set(v, i);
}
return i;
};
const genreSets: number[][] = [];
const genreIndex = new Map<string, number>();
const table: Record<string, unknown> = {
strings,
genreSets,
filePath: tracks.map((t) => t.FilePath),
lengthMs: tracks.map((t) => Number(t['TrackLength'] ?? 0) || 0),
genre: tracks.map((t) => {
const genres = (t['Genre'] as string[] | null | undefined) ?? [];
const key = genres.join('\u0000');
let i = genreIndex.get(key);
if (i === undefined) {
i = genreSets.length;
genreSets.push(genres.map(intern));
genreIndex.set(key, i);
}
return i;
}),
};
for (const [col, field] of STRING_COLS) {
table[col] = tracks.map((t) => intern(t[field] as string | undefined));
}
for (const [col, field] of INT_COLS) {
table[col] = tracks.map((t) => (t[field] as number | undefined) ?? 0);
}
return table as unknown as library.TrackTable;
}
-134
View File
@@ -1,134 +0,0 @@
/**
* `decodeTrackTable` is the only reader of the backend's columnar track
* list (#281). The encoder is Go and the row type is TypeScript, so the
* last block reads both sides' declarations and fails if a column is
* added on one and not the other.
*/
import { describe, expect, it } from 'vitest';
import {
decodeTrackTable,
LIST_TRACK_OMITS,
TrackTableError,
} from '@utils/track-table';
import { trackTable } from '@test/support/track-table';
const FIXTURE = [
{
FilePath: '/m/a/1.flac',
TrackName: 'One',
ArtistName: 'Artist',
Album: 'Album',
Genre: ['Ambient', 'Drone'],
TrackLength: '215000',
TrackNumber: 1,
CoverArtSmall: '/covers/x_sm.jpg',
PlayCount: 3,
},
{
FilePath: '/m/a/2.flac',
TrackName: 'Two',
ArtistName: 'Artist',
Album: 'Album',
Genre: ['Ambient', 'Drone'],
TrackLength: '1000',
TrackNumber: 2,
CoverArtSmall: '/covers/x_sm.jpg',
},
];
describe('decodeTrackTable', () => {
it('gives back each track it was given', () => {
const [a, b] = decodeTrackTable(trackTable(FIXTURE));
expect(a).toMatchObject({
FilePath: '/m/a/1.flac',
TrackName: 'One',
Album: 'Album',
Genre: ['Ambient', 'Drone'],
TrackLength: '215000',
TrackNumber: 1,
PlayCount: 3,
Composer: '',
});
expect(b).toMatchObject({ TrackName: 'Two', TrackLength: '1000', PlayCount: 0 });
});
it('shares one genre list between tracks that have the same one', () => {
const [a, b] = decodeTrackTable(trackTable(FIXTURE));
expect(a!.Genre).toBe(b!.Genre);
});
it('decodes an empty library to no rows', () => {
expect(decodeTrackTable(trackTable([]))).toEqual([]);
});
it('refuses a table whose columns disagree in length', () => {
const broken = trackTable(FIXTURE);
(broken as unknown as { album: number[] }).album = [1];
expect(() => decodeTrackTable(broken)).toThrow(TrackTableError);
});
it('refuses a string index past the table', () => {
const broken = trackTable(FIXTURE);
(broken as unknown as { album: number[] }).album = [1, 999];
expect(() => decodeTrackTable(broken)).toThrow(/out of range/);
});
});
/** The generated Track interface and the Go encoder, as text. */
const MODELS = Object.values(
import.meta.glob<string>('../../bindings/yellowjacket/backend/library/models.ts', {
eager: true,
query: '?raw',
import: 'default',
}),
)[0] ?? '';
const ENCODER = Object.values(
import.meta.glob<string>('../../../backend/library/tracktable.go', {
eager: true,
query: '?raw',
import: 'default',
}),
)[0] ?? '';
function interfaceKeys(source: string, name: string): string[] {
const body = source.split(`export interface ${name} {`)[1]?.split('\n}')[0] ?? '';
return [...body.matchAll(/^\s+"(\w+)"\??:/gm)].map((m) => m[1]!);
}
describe('the list row and the Go table agree', () => {
const trackKeys = interfaceKeys(MODELS, 'Track');
const tableKeys = [...ENCODER.matchAll(/json:"(\w+)"/g)].map((m) => m[1]!);
it('read both declarations', () => {
expect(trackKeys.length).toBeGreaterThan(20);
expect(tableKeys.length).toBeGreaterThan(20);
});
it('decodes every Track field the list keeps, and only those', () => {
const decoded = Object.keys(decodeTrackTable(trackTable(FIXTURE))[0]!).sort();
const omitted = new Set<string>(LIST_TRACK_OMITS);
expect(decoded).toEqual(trackKeys.filter((k) => !omitted.has(k)).sort());
});
it('has a column for every field it decodes', () => {
// Columns are the field names in lower camel case, except the two
// that are not columns of a field and the length, which is sent as
// the number it encodes.
const columns = new Set(tableKeys.filter((k) => !['strings', 'genreSets'].includes(k)));
const expected = trackKeys
.filter((k) => !(LIST_TRACK_OMITS as readonly string[]).includes(k))
.map((k) => (k === 'TrackLength' ? 'lengthMs' : k[0]!.toLowerCase() + k.slice(1)))
.map((k) => k.replace(/MBID$/, 'Mbid'));
expect([...columns].sort()).toEqual(expected.sort());
});
});