onion-mc/analysis/client-perf/harness

Bench harness v2

bench.py drives the OnionBench Prism instance (~/onion-bench/prism/instances/OnionBench) with no input injection: a generated world datapack runs the scenes and prints markers, the harness only reads logs and the MangoHud CSV. run_scene.py is the legacy xdotool harness, kept for reference. Stdlib only, Python 3.

Loop

cd analysis/client-perf/harness
python3 bench.py boot --n 3 --variant C1            # cold launches -> results/boot.jsonl
python3 bench.py boot --n 3 --join                  # ... into the world (join time)
python3 bench.py session --variant C0 --plan s1 --reps 3 --label B0      # x3 sessions for B0
python3 bench.py session --variant C1 --plan s1,s2,s3,s4,s5 --label B2
python3 bench.py session --variant C1 --plan s5 --jvm J2 --heap 6144 --label B4
python3 analyze.py --baseline C0 --noise B0 [--label B2]   # markdown table + verdicts
python3 -m unittest discover -s tests               # offline tests (from this dir)

session --dry-run prints the datapack it would write and changes nothing. bench.py meta prints the environment snapshot; bench.py disarm removes a leftover plan pack.

Per-session instance changes (undone after every session)

python3 bench.py session --variant C1 --plan s1,s2,s4 --shader ComplementaryReimagined_r5.9.3.zip \
    [--shader-options profile.txt] --label B7
python3 bench.py session --variant C1 --plan s1 --dh off
python3 bench.py session --variant C1 --plan s1 --disable-mods 'badoptimizations*,particle_core*'
python3 bench.py session --variant C1 --plan s2 --add-mods ~/trials/gnetum.jar
python3 bench.py session --variant C1 --plan s4 --world-zip ~/onion-bench/worlds/bench-world-v1-s4-1.zip
python3 bench.py restore        # after a hard kill: replay .harness-pending.json

• --shader NAME|off (default off): config/iris.properties enableShaders / shaderPack; NAME must exist in shaderpacks/. --shader-options FILE is copied to shaderpacks/<NAME>.txt (Iris' per-pack options file; journaled whenever a shader is set).

• --dh on|off (default on): [client.advanced.debugging] rendererMode = DEFAULT / DISABLED in config/DistantHorizons.toml, set after the variant (DH's quickEnableRendering is GUI-only and maps to this key).

• --disable-mods GLOB,..: case-insensitive globs over mods/*.jar, renamed to .jar.disabled; a glob that matches nothing is an error. --add-mods PATH,.. copies jars in (name collision = error). mods_hash is taken after these changes.

• --world-zip PATH: saves/onionbench -> saves/onionbench.harness-bak, zip extracted (world at the zip root or under one top directory, needs level.dat); sha256 recorded.

• Every session first resets options.txt resourcePacks / incompatibleResourcePacks to ["vanilla","mod_resources"] / [] (variants may override); not undone.

• Crash safety: before the first change session writes .minecraft/.harness-pending.json (write-ahead: each entry precedes its change; file backups in .minecraft/.harness-backup/). The normal finally and bench.py restore run the same restore, newest entry first; failed entries stay in the journal. session and boot refuse to start while it exists. SIGTERM unwinds through the restore; SIGKILL leaves the journal for bench.py restore.

• Rows record shader, shader_options(_sha256), dh, disabled_mods, added_mods, world_zip(_sha256), resource_packs, restore_errors; warn flags gpu_dpm_auto (RX 7800 XT power_dpm_force_performance_level != high), server_running, restore_failed.

• boot: N launches; waits for Game took N seconds (or joined the game with --join); outcome ok / crash / timeout. Crash = new file in crash-reports/ or hs_err_pid*.log, client java exited, or a log line with Failed to create mod instance, a /FATAL] level tag, Cowardly refusing, or a crash-report header.

• session: applies the variant (proposed-configs/apply_variant.py), then pins options.txt (no onboarding/narrator, no pause on focus loss, maxFps:260, vsync off), hashes config/ (minus .bak/.tmp/spark buffers) and options.txt, writes the plan datapack, launches once into onionbench, tails latest.log until BENCH_DONE, crash, exit or timeout, kills the client, removes the plan pack, then slices the CSV. One JSON row per scene x rep window goes to ~/onion-bench/results/session.jsonl; every session's latest.log is archived under ~/onion-bench/results/logs/.

• Only java processes whose command line contains instances/OnionBench are ever signalled. The dedicated server (unix_args.txt) is only observed (running + %CPU).

• --jvm J0..J3 / --heap MB rewrite JvmArgs / MaxMemAlloc in the bench instance.cfg (first original saved as instance.cfg.harness-orig). Without them the instance is left as is; the effective -X* flags are read from the running client.

Datapack mechanism

The scene functions live in the permanent pack saves/onionbench/datapacks/onion-bench (bench:s1 ... bench:tour_start, source in ../bench/onion-bench). Per session the harness writes datapacks/onion-bench-autoplan/ (pack_format 48, 1.21.1 function/ dirs) with a session id pYYYYMMDDHHMMSS:

• #minecraft:load -> bench:autoplan/<sid>/load: unless score #<sid> bench_plan is set, poll every second (schedule) until execute if entity @a;
• begin: set the score (a /reload won't restart the plan), clear stale storm_tick / tour schedules, say BENCH_PLAN <sid> begin, wait --settle s;
• chained stepN functions: execute as @a[limit=1] at @s run function bench:<scene>, say BENCH_MARK warm|start|end <scene> <rep>, schedule the warm-up / capture ticks; say BENCH_DONE <sid> at the end.

S5 is fixed by BENCH-MATRIX: bench:s1, 10 s, bench:tour_start, all 120 s captured (warm-up ignored). S2s = S2 + bench:storm_on during warm-up and capture. Plans must follow S1 -> S5 order (--any-order overrides). Function names contain the session id, so schedules saved into level.dat by an autosave can never fire in a later session. The pack is deleted in a finally, so normal play of the world is unaffected; after a hard kill of the harness itself run bench.py disarm.

Markers are taken from the server log line ([Server] BENCH_MARK ..., ms timestamps). The MangoHud CSV used is one created at/after launch (never a stale one); wall time = file birth time (stat -c %.9W, else the name's timestamp) + elapsed, and the method is recorded as align. Per window: n, p50/p95/p99/peak ms, avg fps (1000/mean frametime), 1 % low = 1000/mean of the worst 1 % frametimes, and the legacy fps-column 1 % low.

Flags: vsync_suspect (p50 within 3 % of 60 Hz or an xrandr refresh), low_n (n < 500 or < half of duration x avg fps), crashed/timeout (window never ended), session_crash (window ended, session crashed later), backup_in_window (backup or autosave line from 5 s before start to end; warn only). analyze.py drops rows with any non-warn flag unless --include-invalid.

Metadata per row: label, machine (proxy-9900X-7800XT), git sha + dirty, harness sha, variant + config/options hashes, mods hash/count, Java version, cfg JVM args + effective flags, MaxMemAlloc, kernel, CPU governor, per-GPU DPM level + sclk, MangoHud's GPU string, MemAvailable before/after, load average, server running + %CPU, startup and join seconds, block-atlas size, level.dat hash.

Known limits

• The desktop is KDE Plasma on Wayland (kwin_wayland + Xwayland at :1). xdotool only reaches Xwayland windows, windowactivate --sync hangs under kwin, focus-steal prevention can drop keystrokes, and typed chat lands in whatever has focus. Hence no input injection at all in v2.
• Still manual: S6 (JEI/GUI overlay needs a key press), producing B7 shader option profiles (export once from Iris' GUI, then pass --shader-options), checks needing F3 reads, /sparkc heap/GC reads (B1/B4).
• MangoHud's log_duration=1800 in ~/.local/bin/mangohud-local caps one CSV at 30 min; session warns when a plan is longer (B5's 45-min decay run needs a split or a larger value).
• The integrated server ticks the plan, so scene timing stretches if it lags; windows use the marker wall times, not tick counts. The marker chat line is visible for ~10 s at the start of each capture (identical across variants).
• FTB Backups 3 backs up the world right after join (its index ftbbackups3/backups.json doesn't parse, so it treats the world as never backed up) and every backup_timer = 120 min. Done (bench instance only): auto: false in the instance's config/ftbbackups3-server.snbt. Never edit the repo's config/ for this.
• Stop the dedicated server (tmux onion-mc) while benching (owner-approved); it used up to ~78 % of a core idle. Rows are flagged server_running if it is up.
• amdgpu DPM is auto (no root); rows are flagged gpu_dpm_auto. For tighter noise: echo high | sudo tee /sys/class/drm/card1/device/power_dpm_force_performance_level.
• Scenes run in spectator mode (set by the plan) so elevated cameras don't fall.