logan
a15c912ad0
- bench.py: gamescope-headless display (default, 1920x1080; ~2% cost vs desktop for isolation from the shared Wayland desktop), --shader, --dh, --disable-mods/--add-mods, --world-zip, crash-safe restore journal, spectator mode in the plan (elevated cameras fell to the sea floor and S4 measured underwater darkness) - matrix.toml/matrix.py: declarative, resumable B0-B14 runner keyed on mods hash (warns when B0 must be re-run) - build.py: bench-only resource-pack variants (C1-fa, C1-fa-ext, emf-lod*-fa, C1-bl); every bench variant pins resourcePacks so packs don't leak between sessions; shipped tier output unchanged - analyze.py: tagged labels use their own block's baseline
140 lines
9.0 KiB
Markdown
140 lines
9.0 KiB
Markdown
# Bench harness v2
|
|
|
|
`bench.py` drives the OnionBench Prism instance (`~/onion-bench/prism/instances/OnionBench`)
|
|
with **no input injection**: a generated world datapack runs the scenes and prints
|
|
markers, the harness only reads logs and the MangoHud CSV. `run_scene.py` is the legacy
|
|
xdotool harness, kept for reference. Stdlib only, Python 3.
|
|
|
|
## Loop
|
|
|
|
```sh
|
|
cd analysis/client-perf/harness
|
|
python3 bench.py boot --n 3 --variant C1 # cold launches -> results/boot.jsonl
|
|
python3 bench.py boot --n 3 --join # ... into the world (join time)
|
|
python3 bench.py session --variant C0 --plan s1 --reps 3 --label B0 # x3 sessions for B0
|
|
python3 bench.py session --variant C1 --plan s1,s2,s3,s4,s5 --label B2
|
|
python3 bench.py session --variant C1 --plan s5 --jvm J2 --heap 6144 --label B4
|
|
python3 analyze.py --baseline C0 --noise B0 [--label B2] # markdown table + verdicts
|
|
python3 -m unittest discover -s tests # offline tests (from this dir)
|
|
```
|
|
|
|
`session --dry-run` prints the datapack it would write and changes nothing. `bench.py meta`
|
|
prints the environment snapshot; `bench.py disarm` removes a leftover plan pack.
|
|
|
|
## Per-session instance changes (undone after every session)
|
|
|
|
```sh
|
|
python3 bench.py session --variant C1 --plan s1,s2,s4 --shader ComplementaryReimagined_r5.9.3.zip \
|
|
[--shader-options profile.txt] --label B7
|
|
python3 bench.py session --variant C1 --plan s1 --dh off
|
|
python3 bench.py session --variant C1 --plan s1 --disable-mods 'badoptimizations*,particle_core*'
|
|
python3 bench.py session --variant C1 --plan s2 --add-mods ~/trials/gnetum.jar
|
|
python3 bench.py session --variant C1 --plan s4 --world-zip ~/onion-bench/worlds/bench-world-v1-s4-1.zip
|
|
python3 bench.py restore # after a hard kill: replay .harness-pending.json
|
|
```
|
|
|
|
- `--shader NAME|off` (default `off`): `config/iris.properties` `enableShaders` /
|
|
`shaderPack`; NAME must exist in `shaderpacks/`. `--shader-options FILE` is copied to
|
|
`shaderpacks/<NAME>.txt` (Iris' per-pack options file; journaled whenever a shader is set).
|
|
- `--dh on|off` (default `on`): `[client.advanced.debugging] rendererMode` =
|
|
`DEFAULT` / `DISABLED` in `config/DistantHorizons.toml`, set after the variant (DH's
|
|
`quickEnableRendering` is GUI-only and maps to this key).
|
|
- `--disable-mods GLOB,..`: case-insensitive globs over `mods/*.jar`, renamed to
|
|
`.jar.disabled`; a glob that matches nothing is an error. `--add-mods PATH,..` copies jars
|
|
in (name collision = error). `mods_hash` is taken after these changes.
|
|
- `--world-zip PATH`: `saves/onionbench` -> `saves/onionbench.harness-bak`, zip extracted
|
|
(world at the zip root or under one top directory, needs `level.dat`); sha256 recorded.
|
|
- Every session first resets `options.txt` `resourcePacks` / `incompatibleResourcePacks`
|
|
to `["vanilla","mod_resources"]` / `[]` (variants may override); not undone.
|
|
- Crash safety: before the first change `session` writes `.minecraft/.harness-pending.json`
|
|
(write-ahead: each entry precedes its change; file backups in `.minecraft/.harness-backup/`).
|
|
The normal `finally` and `bench.py restore` run the same restore, newest entry first;
|
|
failed entries stay in the journal. `session` and `boot` refuse to start while it exists.
|
|
SIGTERM unwinds through the restore; SIGKILL leaves the journal for `bench.py restore`.
|
|
- Rows record `shader`, `shader_options(_sha256)`, `dh`, `disabled_mods`, `added_mods`,
|
|
`world_zip(_sha256)`, `resource_packs`, `restore_errors`; warn flags `gpu_dpm_auto` (RX
|
|
7800 XT `power_dpm_force_performance_level` != `high`), `server_running`,
|
|
`restore_failed`.
|
|
|
|
- **boot**: N launches; waits for `Game took N seconds` (or `joined the game` with
|
|
`--join`); outcome `ok` / `crash` / `timeout`. Crash = new file in `crash-reports/` or
|
|
`hs_err_pid*.log`, client java exited, or a log line with `Failed to create mod
|
|
instance`, a `/FATAL]` level tag, `Cowardly refusing`, or a crash-report header.
|
|
- **session**: applies the variant (`proposed-configs/apply_variant.py`), then pins
|
|
`options.txt` (no onboarding/narrator, no pause on focus loss, `maxFps:260`, vsync
|
|
off), hashes `config/` (minus `.bak`/`.tmp`/spark buffers) and `options.txt`, writes the
|
|
plan datapack, launches once into `onionbench`, tails `latest.log` until `BENCH_DONE`,
|
|
crash, exit or timeout, kills the client, removes the plan pack, then slices the CSV.
|
|
One JSON row per scene x rep window goes to `~/onion-bench/results/session.jsonl`;
|
|
every session's `latest.log` is archived under `~/onion-bench/results/logs/`.
|
|
- Only java processes whose command line contains `instances/OnionBench` are ever
|
|
signalled. The dedicated server (`unix_args.txt`) is only observed (running + %CPU).
|
|
- `--jvm J0..J3` / `--heap MB` rewrite `JvmArgs` / `MaxMemAlloc` in the bench
|
|
`instance.cfg` (first original saved as `instance.cfg.harness-orig`). Without them the
|
|
instance is left as is; the effective `-X*` flags are read from the running client.
|
|
|
|
## Datapack mechanism
|
|
|
|
The scene functions live in the permanent pack `saves/onionbench/datapacks/onion-bench`
|
|
(`bench:s1` ... `bench:tour_start`, source in `../bench/onion-bench`). Per session the
|
|
harness writes `datapacks/onion-bench-autoplan/` (pack_format 48, 1.21.1 `function/`
|
|
dirs) with a session id `pYYYYMMDDHHMMSS`:
|
|
|
|
- `#minecraft:load` -> `bench:autoplan/<sid>/load`: unless score `#<sid> bench_plan` is
|
|
set, poll every second (`schedule`) until `execute if entity @a`;
|
|
- `begin`: set the score (a `/reload` won't restart the plan), clear stale
|
|
`storm_tick` / tour schedules, `say BENCH_PLAN <sid> begin`, wait `--settle` s;
|
|
- chained `stepN` functions: `execute as @a[limit=1] at @s run function bench:<scene>`,
|
|
`say BENCH_MARK warm|start|end <scene> <rep>`, `schedule` the warm-up / capture ticks;
|
|
`say BENCH_DONE <sid>` at the end.
|
|
|
|
S5 is fixed by BENCH-MATRIX: `bench:s1`, 10 s, `bench:tour_start`, all 120 s captured
|
|
(warm-up ignored). S2s = S2 + `bench:storm_on` during warm-up and capture. Plans must
|
|
follow S1 -> S5 order (`--any-order` overrides). Function names contain the session id, so
|
|
schedules saved into `level.dat` by an autosave can never fire in a later session. The pack
|
|
is deleted in a `finally`, so normal play of the world is unaffected; after a hard kill
|
|
of the harness itself run `bench.py disarm`.
|
|
|
|
Markers are taken from the server log line (`[Server] BENCH_MARK ...`, ms timestamps).
|
|
The MangoHud CSV used is one created at/after launch (never a stale one); wall time =
|
|
file birth time (`stat -c %.9W`, else the name's timestamp) + `elapsed`, and the method is
|
|
recorded as `align`. Per window: n, p50/p95/p99/peak ms, avg fps (1000/mean frametime),
|
|
1 % low = 1000/mean of the worst 1 % frametimes, and the legacy fps-column 1 % low.
|
|
|
|
Flags: `vsync_suspect` (p50 within 3 % of 60 Hz or an xrandr refresh), `low_n`
|
|
(n < 500 or < half of duration x avg fps), `crashed`/`timeout` (window never ended),
|
|
`session_crash` (window ended, session crashed later), `backup_in_window` (backup or
|
|
autosave line from 5 s before start to end; warn only). `analyze.py` drops rows with
|
|
any non-warn flag unless `--include-invalid`.
|
|
|
|
Metadata per row: label, machine (`proxy-9900X-7800XT`), git sha + dirty, harness sha,
|
|
variant + config/options hashes, mods hash/count, Java version, cfg JVM args + effective
|
|
flags, MaxMemAlloc, kernel, CPU governor, per-GPU DPM level + sclk, MangoHud's GPU
|
|
string, MemAvailable before/after, load average, server running + %CPU, startup and
|
|
join seconds, block-atlas size, `level.dat` hash.
|
|
|
|
## Known limits
|
|
|
|
- The desktop is **KDE Plasma on Wayland** (kwin_wayland + Xwayland at `:1`). xdotool
|
|
only reaches Xwayland windows, `windowactivate --sync` hangs under kwin, focus-steal
|
|
prevention can drop keystrokes, and typed chat lands in whatever has focus. Hence no
|
|
input injection at all in v2.
|
|
- Still manual: **S6** (JEI/GUI overlay needs a key press), producing B7 shader option
|
|
profiles (export once from Iris' GUI, then pass `--shader-options`), checks needing F3
|
|
reads, `/sparkc` heap/GC reads (B1/B4).
|
|
- MangoHud's `log_duration=1800` in `~/.local/bin/mangohud-local` caps one CSV at 30 min;
|
|
`session` warns when a plan is longer (B5's 45-min decay run needs a split or a larger
|
|
value).
|
|
- The integrated server ticks the plan, so scene timing stretches if it lags; windows use
|
|
the marker wall times, not tick counts. The marker chat line is visible for ~10 s at
|
|
the start of each capture (identical across variants).
|
|
- **FTB Backups 3** backs up the world right after join (its index
|
|
`ftbbackups3/backups.json` doesn't parse, so it treats the world as never backed up)
|
|
and every `backup_timer` = 120 min. **Done (bench instance only):** `auto: false` in
|
|
the instance's `config/ftbbackups3-server.snbt`. Never edit the repo's `config/` for this.
|
|
- Stop the dedicated server (tmux `onion-mc`) while benching (owner-approved); it used up
|
|
to ~78 % of a core idle. Rows are flagged `server_running` if it is up.
|
|
- amdgpu DPM is `auto` (no root); rows are flagged `gpu_dpm_auto`. For tighter noise:
|
|
`echo high | sudo tee /sys/class/drm/card1/device/power_dpm_force_performance_level`.
|
|
- Scenes run in **spectator** mode (set by the plan) so elevated cameras don't fall.
|