onion-mc/analysis/client-perf/harness/README.md

Bench harness v2

bench.py drives the OnionBench Prism instance (~/onion-bench/prism/instances/OnionBench) with no input injection: a generated world datapack runs the scenes and prints markers, the harness only reads logs and the MangoHud CSV. run_scene.py is the legacy xdotool harness, kept for reference. Stdlib only, Python 3.

Loop

cd analysis/client-perf/harness
python3 bench.py boot --n 3 --variant C1            # cold launches -> results/boot.jsonl
python3 bench.py boot --n 3 --join                  # ... into the world (join time)
python3 bench.py session --variant C0 --plan s1 --reps 3 --label B0      # x3 sessions for B0
python3 bench.py session --variant C1 --plan s1,s2,s3,s4,s5 --label B2
python3 bench.py session --variant C1 --plan s5 --jvm J2 --heap 6144 --label B4
python3 analyze.py --baseline C0 --noise B0 [--label B2]   # markdown table + verdicts
python3 -m unittest discover -s tests               # offline tests (from this dir)

session --dry-run prints the datapack it would write and changes nothing. bench.py meta prints the environment snapshot; bench.py disarm removes a leftover plan pack.

• boot: N launches; waits for Game took N seconds (or joined the game with --join); outcome ok / crash / timeout. Crash = new file in crash-reports/ or hs_err_pid*.log, client java exited, or a log line with Failed to create mod instance, a /FATAL] level tag, Cowardly refusing, or a crash-report header.
• session: applies the variant (proposed-configs/apply_variant.py), then pins options.txt (no onboarding/narrator, no pause on focus loss, maxFps:260, vsync off), hashes config/ (minus .bak/.tmp/spark buffers) and options.txt, writes the plan datapack, launches once into onionbench, tails latest.log until BENCH_DONE, crash, exit or timeout, kills the client, removes the plan pack, then slices the CSV. One JSON row per scene x rep window goes to ~/onion-bench/results/session.jsonl; every session's latest.log is archived under ~/onion-bench/results/logs/.
• Only java processes whose command line contains instances/OnionBench are ever signalled. The dedicated server (unix_args.txt) is only observed (running + %CPU).
• --jvm J0..J3 / --heap MB rewrite JvmArgs / MaxMemAlloc in the bench instance.cfg (first original saved as instance.cfg.harness-orig). Without them the instance is left as is; the effective -X* flags are read from the running client.

Datapack mechanism

The scene functions live in the permanent pack saves/onionbench/datapacks/onion-bench (bench:s1 ... bench:tour_start, source in ../bench/onion-bench). Per session the harness writes datapacks/onion-bench-autoplan/ (pack_format 48, 1.21.1 function/ dirs) with a session id pYYYYMMDDHHMMSS:

• #minecraft:load -> bench:autoplan/<sid>/load: unless score #<sid> bench_plan is set, poll every second (schedule) until execute if entity @a;
• begin: set the score (a /reload won't restart the plan), clear stale storm_tick / tour schedules, say BENCH_PLAN <sid> begin, wait --settle s;
• chained stepN functions: execute as @a[limit=1] at @s run function bench:<scene>, say BENCH_MARK warm|start|end <scene> <rep>, schedule the warm-up / capture ticks; say BENCH_DONE <sid> at the end.

S5 is fixed by BENCH-MATRIX: bench:s1, 10 s, bench:tour_start, all 120 s captured (warm-up ignored). S2s = S2 + bench:storm_on during warm-up and capture. Plans must follow S1 -> S5 order (--any-order overrides). Function names contain the session id, so schedules saved into level.dat by an autosave can never fire in a later session. The pack is deleted in a finally, so normal play of the world is unaffected; after a hard kill of the harness itself run bench.py disarm.

Markers are taken from the server log line ([Server] BENCH_MARK ..., ms timestamps). The MangoHud CSV used is one created at/after launch (never a stale one); wall time = file birth time (stat -c %.9W, else the name's timestamp) + elapsed, and the method is recorded as align. Per window: n, p50/p95/p99/peak ms, avg fps (1000/mean frametime), 1 % low = 1000/mean of the worst 1 % frametimes, and the legacy fps-column 1 % low.

Flags: vsync_suspect (p50 within 3 % of 60 Hz or an xrandr refresh), low_n (n < 500 or < half of duration x avg fps), crashed/timeout (window never ended), session_crash (window ended, session crashed later), backup_in_window (backup or autosave line from 5 s before start to end; warn only). analyze.py drops rows with any non-warn flag unless --include-invalid.

Metadata per row: label, machine (proxy-9900X-7800XT), git sha + dirty, harness sha, variant + config/options hashes, mods hash/count, Java version, cfg JVM args + effective flags, MaxMemAlloc, kernel, CPU governor, per-GPU DPM level + sclk, MangoHud's GPU string, MemAvailable before/after, load average, server running + %CPU, startup and join seconds, block-atlas size, level.dat hash.

Known limits

• The desktop is KDE Plasma on Wayland (kwin_wayland + Xwayland at :1). xdotool only reaches Xwayland windows, windowactivate --sync hangs under kwin, focus-steal prevention can drop keystrokes, and typed chat lands in whatever has focus. Hence no input injection at all in v2.
• Still manual: S6 (JEI/GUI overlay needs a key press), B7 shader profiles (set in Iris' GUI), dh-off-style checks needing F3 reads, /sparkc heap/GC reads (B1/B4).
• MangoHud's log_duration=1800 in ~/.local/bin/mangohud-local caps one CSV at 30 min; session warns when a plan is longer (B5's 45-min decay run needs a split or a larger value).
• The integrated server ticks the plan, so scene timing stretches if it lags; windows use the marker wall times, not tick counts. The marker chat line is visible for ~10 s at the start of each capture (identical across variants).
• FTB Backups 3 backs up the world right after join (its index ftbbackups3/backups.json doesn't parse, so it treats the world as never backed up) and every backup_timer = 120 min. For the bench instance only, set auto: false in saves/onionbench/serverconfig/ftbbackups3-server.snbt (copy of config/ftbbackups3-server.snbt; the world override wins and is never touched by pack updates). Not done by the harness. Never edit the repo's config/ for this.
• The dedicated server in tmux onion-mc keeps running during benches and was observed using ~78 % of a core while idle; it is recorded per row as a noise covariate.