yonlu/yellowjacket
1
1
Fork 0
Code Issues 55 Pull Requests 6 Actions Packages Projects Releases 12 Wiki Activity

Android: SQLite has no temp directory, so the champion index never builds #190

New Issue
Closed
opened 2026-08-21 20:17:52 +00:00 by logan · 1 comment
logan commented 2026-08-21 20:17:52 +00:00
Collaborator
Copy Link
Copy Source

Report

The champion search index never builds on Android. It fails with a
SQLite I/O error whose extended code names the cause exactly:
there is no temp directory.

W/yellowjacket: msg="champion index rebuild failed"
  explore.search-index.error="populate champion fts: disk I/O error (6410)"

Measured on the reference device (TLP301, Android 14, arm64) with a
1,577-track library and a populated catalog (search index: using existing index entries=1079667). Found in the first minute of logcat
after #160 landed; it is written on every launch and was going to
/dev/null before that.

Findings

6410 is not a generic I/O error. SQLite's extended result codes pack
the detail into the high bits:

6410 & 0xff == 10     -> SQLITE_IOERR
6410 >> 8   == 25     -> SQLITE_IOERR_GETTEMPPATH  (10 | (25<<8))

SQLite could not work out where to put a temporary file. Two
measurements on the device say why:

$ adb shell ls -d /tmp
ls: /tmp: No such file or directory

$ adb shell run-as app.yellowjacket.dev cat /proc/<pid>/environ | grep -i TMP
(nothing)

Android has no /tmp, and the app's process has no TMPDIR. The
zygote does not set one — a shell does (/data/local/tmp), which is
why this is easy to miss from adb shell. Go's os.TempDir() falls
back to /tmp when TMPDIR is unset, so the pure-Go SQLite driver
asks for a temp path, is handed a directory that does not exist, and
returns GETTEMPPATH.

The statement that trips it is the champion FTS populate in
backend/explore/searchindex.go:

INSERT INTO explore_champion_fts(rowid, title, artist_name, aliases)
SELECT id, title, artist_name, aliases
FROM explore_index
WHERE popularity >= ? OR in_library = 1

An INSERT ... SELECT of that size against an FTS table spills, which
is when a temp file is wanted. So the trigger is the size of the
work, not this statement in particular — anything that spills fails
the same way on this platform, which makes this a class rather than
one query. Large sorts, large joins and VACUUM are the usual others.

What it costs

championReady stays false, so every Explore search on Android takes
the generic path over the whole 1,079,667-row index instead of the
champion subset the rebuild exists to produce. It is a silent
performance cliff on the slowest device this app runs on, and it
re-attempts on every launch, so the cost is paid repeatedly.

Direction

Two candidate fixes, and they are not equivalent:

  1. Set TMPDIR for the process on Android, from the app's own
    cache dir, next to where system.UseHomeOverride(application.Mobile .StoragePath()) already runs in main.go. This fixes the class
    rather than the statement, and it fixes it for anything else in the
    process that calls os.TempDir(). It needs a real writable
    directory the app owns — getCacheDir() is the Android answer, and
    whether Wails exposes it needs checking (Mobile.StoragePath() is
    getFilesDir(), which would work but is the durable directory, not
    a cache).
  2. PRAGMA temp_store = MEMORY on the connection, which stops
    SQLite wanting a file at all. Cheaper and more local, but it is a
    promise about every future spill fitting in RAM on a phone, and
    the catalog is the largest thing in this app. It also silently
    changes the memory profile of statements nobody is thinking about.

(1) is the better default and (2) is worth considering in addition
for the catalog specifically. Whichever is chosen, the check is that
champion index rebuilt appears in make android-logs where
champion index rebuild failed is today.

Worth knowing

No tier here can see this: it is a property of the process environment
on one platform, and CI, the e2e harness and every desktop run have a
/tmp. The diagnosis is entirely in the extended result code, so
whatever logs a SQLite error should keep logging the number — a
message reading only "disk I/O error" would have led nowhere.

**Report** The champion search index never builds on Android. It fails with a SQLite I/O error whose extended code names the cause exactly: **there is no temp directory**. ``` W/yellowjacket: msg="champion index rebuild failed" explore.search-index.error="populate champion fts: disk I/O error (6410)" ``` Measured on the reference device (TLP301, Android 14, arm64) with a 1,577-track library and a populated catalog (`search index: using existing index entries=1079667`). Found in the first minute of logcat after #160 landed; it is written on every launch and was going to `/dev/null` before that. **Findings** `6410` is not a generic I/O error. SQLite's extended result codes pack the detail into the high bits: ``` 6410 & 0xff == 10 -> SQLITE_IOERR 6410 >> 8 == 25 -> SQLITE_IOERR_GETTEMPPATH (10 | (25<<8)) ``` SQLite could not work out where to put a temporary file. Two measurements on the device say why: ``` $ adb shell ls -d /tmp ls: /tmp: No such file or directory $ adb shell run-as app.yellowjacket.dev cat /proc/<pid>/environ | grep -i TMP (nothing) ``` **Android has no `/tmp`, and the app's process has no `TMPDIR`.** The zygote does not set one — a shell does (`/data/local/tmp`), which is why this is easy to miss from `adb shell`. Go's `os.TempDir()` falls back to `/tmp` when `TMPDIR` is unset, so the pure-Go SQLite driver asks for a temp path, is handed a directory that does not exist, and returns GETTEMPPATH. The statement that trips it is the champion FTS populate in `backend/explore/searchindex.go`: ```sql INSERT INTO explore_champion_fts(rowid, title, artist_name, aliases) SELECT id, title, artist_name, aliases FROM explore_index WHERE popularity >= ? OR in_library = 1 ``` An `INSERT ... SELECT` of that size against an FTS table spills, which is when a temp file is wanted. So the trigger is the *size* of the work, not this statement in particular — **anything that spills fails the same way on this platform**, which makes this a class rather than one query. Large sorts, large joins and `VACUUM` are the usual others. **What it costs** `championReady` stays false, so every Explore search on Android takes the generic path over the whole 1,079,667-row index instead of the champion subset the rebuild exists to produce. It is a silent performance cliff on the slowest device this app runs on, and it re-attempts on every launch, so the cost is paid repeatedly. **Direction** Two candidate fixes, and they are not equivalent: 1. **Set `TMPDIR` for the process on Android**, from the app's own cache dir, next to where `system.UseHomeOverride(application.Mobile .StoragePath())` already runs in `main.go`. This fixes the class rather than the statement, and it fixes it for anything else in the process that calls `os.TempDir()`. It needs a real writable directory the app owns — `getCacheDir()` is the Android answer, and whether Wails exposes it needs checking (`Mobile.StoragePath()` is `getFilesDir()`, which would work but is the durable directory, not a cache). 2. **`PRAGMA temp_store = MEMORY`** on the connection, which stops SQLite wanting a file at all. Cheaper and more local, but it is a promise about *every* future spill fitting in RAM on a phone, and the catalog is the largest thing in this app. It also silently changes the memory profile of statements nobody is thinking about. (1) is the better default and (2) is worth considering *in addition* for the catalog specifically. Whichever is chosen, the check is that `champion index rebuilt` appears in `make android-logs` where `champion index rebuild failed` is today. **Worth knowing** No tier here can see this: it is a property of the process environment on one platform, and CI, the e2e harness and every desktop run have a `/tmp`. The diagnosis is entirely in the extended result code, so whatever logs a SQLite error should keep logging the number — a message reading only "disk I/O error" would have led nowhere.
logan self-assigned this 2026-08-21 21:12:15 +00:00
logan added the
Status
In Progress
 label 2026-08-21 21:12:15 +00:00
logan commented 2026-08-21 21:12:15 +00:00
Author
Collaborator
Copy Link
Copy Source

Taking this together with #189 on 189-190-explore-correctness.

Going with direction (1), TMPDIR, over PRAGMA temp_store = MEMORY:
it fixes the class rather than the statement, and it covers anything
else in the process that calls os.TempDir().

It also needs no new Wails API, which is a correction to what I wrote
above. The issue says a cache dir is wanted and that whether Wails
exposes one needs checking. It does not need one:
system.UseHomeOverride(application.Mobile.StoragePath()) already runs
in main.go and gives an app-owned writable directory, so TMPDIR can
point at a subdirectory of YJ_HOME with no Java change and nothing new
from the runtime.

Taking this together with #189 on `189-190-explore-correctness`. Going with direction (1), `TMPDIR`, over `PRAGMA temp_store = MEMORY`: it fixes the class rather than the statement, and it covers anything else in the process that calls `os.TempDir()`. **It also needs no new Wails API, which is a correction to what I wrote above.** The issue says a cache dir is wanted and that whether Wails exposes one needs checking. It does not need one: `system.UseHomeOverride(application.Mobile.StoragePath())` already runs in `main.go` and gives an app-owned writable directory, so TMPDIR can point at a subdirectory of YJ_HOME with no Java change and nothing new from the runtime.
logan closed this issue 2026-08-21 21:43:38 +00:00
gitea-actions bot removed the
Status
In Progress
 label 2026-08-21 21:43:56 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Area/Design
Area/Downloads
Area/Explore
Area/Library-UI
Area/Metadata
Area/Packaging
Area/Player
Area/Queue
Area/Settings
Area/Shell-Nav
Compat/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
 Platform/Android
Platform/Desktop
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Medium
3
The priority is medium
Priority
Low
4
The priority is low
Reviewed
Confirmed
1
Issue has been confirmed
Reviewed
Duplicate
2
This issue or pull request already exists
Reviewed
Invalid
3
Invalid issue
Reviewed
Won't Fix
3
This issue won't be fixed
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status
In Progress
Somebody is actively working on this right now
No labels Area/Explore Kind/Bug Platform/Android
Priority
High
2
Reviewed
Confirmed
1
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
logan yonlu
No Assignees logan
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: yonlu/yellowjacket#190