yonluandClaude Opus 5
590a0d86dd
Every parser in `backend/metadata` is header-only -- a few hundred bytes and return -- so on a spinning disk a scan is not waiting on CPU or on bytes, it is waiting on the head to arrive. Two things follow, and the drive says which. **How many reads should be in flight.** This was a flat 2 for anything rotational, which is a pre-NCQ assumption: a modern SATA disk reports a queue depth of 32 and reorders outstanding reads into the order its head passes over them, and was being handed a quarter of what it can use. It gets 4 now. A drive that reports 1 -- a USB bridge, a pre-2004 disk -- services one command at a time in the order given, where every extra worker is one more seek competing for one head and the scan gets *slower* the harder it is pushed; that keeps 2. **And that the next seek should already be queued.** A prefetch stage between the walk and the workers issues `POSIX_FADV_WILLNEED` over the first 512 KB of each file -- enough for an ID3v2 tag carrying cover art, or FLAC's STREAMINFO and PICTURE blocks. The buffered channel *is* the lookahead: the goroutine runs 16 files ahead of the workers, hinting as it goes, so the read a worker needs has been in flight for sixteen files' worth of parsing by the time it asks. Rotational only; an SSD gets the channel back unwrapped and pays nothing, since it has no seek to hide and already has one worker per core. `workersForProfile` is the policy on its own so it can be tested against drives this machine does not have, and the scan logs the device, its rotational flag and its queue depth, so the decision is inspectable rather than inferred. Also: `ScanConcurrency` has been a validated three-value config field with exactly one caller, passing the constant `auto` -- so choosing `ssd` or `hdd` by hand did nothing at all. It reads the config now. The two modes overrule detection about the *disk* and not about its queue, since a user who picks `hdd` on a queueing drive still wants that drive's queue used. What is not here is inode-ordered dispatch. It needs the streaming walk restructured to buffer per directory, and with queueing the drive is already reordering what the hints put in front of it; that wants a measurement on real hardware before the complexity. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MeQt5hgXg5YGoNZQ9ozG7L
39 lines
1.1 KiB
Go
39 lines
1.1 KiB
Go
//go:build linux
|
|
|
|
package library
|
|
|
|
import (
|
|
"os"
|
|
|
|
"golang.org/x/sys/unix"
|
|
)
|
|
|
|
// hintReadahead asks the kernel to start fetching the head of a file
|
|
// that is about to be read.
|
|
//
|
|
// `POSIX_FADV_WILLNEED` returns immediately and queues the read, which
|
|
// is the whole point: on a spinning disk the first access to a file
|
|
// costs a seek of several milliseconds, and that latency can only be
|
|
// hidden by having the next seek already in flight while the current
|
|
// file is being parsed. A drive with command queueing can then service
|
|
// the queued reads in head order rather than in the order they were
|
|
// asked for.
|
|
//
|
|
// Errors are dropped on purpose. This is a hint: a file that has since
|
|
// been deleted, a filesystem that does not implement fadvise, or a
|
|
// permission the walk saw and this open does not, all mean "no
|
|
// prefetch", never "fail the scan". The read that follows is what
|
|
// reports a genuine problem.
|
|
func hintReadahead(path string, bytes int64) {
|
|
f, err := os.Open(path)
|
|
if err != nil {
|
|
return
|
|
}
|
|
|
|
defer func() { _ = f.Close() }()
|
|
|
|
_ = unix.Fadvise(
|
|
int(f.Fd()), 0, bytes, unix.FADV_WILLNEED,
|
|
)
|
|
}
|