chore(bench): add client bench harness and Phase 2 handoff

The client now boots cleanly on the bench machine (the CAC spawn-key
crash was fixed in 8e8483d). Add the harness used to drive bench scenes
and capture frametimes, the bench-world generation workflow, and a
self-contained handoff brief. All files live under analysis/ (packwizignored),
so nothing player-facing changes.

- harness/rcon.py: minimal RCON client for the temp worldgen server
- harness/run_scene.py: apply variant -> launch Prism under MangoHud ->
  xdotool scene commands -> p50/p95/p99/1%-low from the CSV
- HANDOFF.md: state, gotchas, next steps
- build.py: disable DH autoUpdater for bench variants (suppress modal)
This commit is contained in:
Onion Agent committed 2026-10-14 23:16:31 -04:00
1 parent 8e8483d072
commit e5e369a693
7 files changed
+611 -1

No files matched your search

@@ -0,0 +1,57 @@
# Phase 0 distill — capture protocol, sheet columns, per-machine prep, LEVERS §7 corrections
Sources: `phase0/P0-steamdeck.md`, `phase0/P1-windows-7700k-1060.md`, `phase0/P2-linux-6700k-1080ti.md`, `LEVERS.md` §7.
## (a) Benchmark capture procedure (shared protocol, all machines)
- [ ] **Uncapped, vsync OFF during measurement.** Apply the FPS cap only *after* tuning; with a cap/vsync every config "hits" the target.
- [ ] Cap/vsync settings: in-game **Max framerate: Unlimited**, **VSync: Off**; MangoHud `fps_limit=0`; SteamOS **Framerate Limit → Off**; P1: NVIDIA panel Vertical sync → Off too (**two vsync switches**, check both each session); P2: compositor vsync disabled.
- [ ] **Metric: p99 frametime + 1% low FPS, not average FPS** (sheet also logs p50/p95, avg FPS, peak frametime).
- [ ] **Discard 10–15 s of warmup per scene** (JIT, chunk load, initial GC).
- [ ] **Settle after any render-distance or DH change** — both trigger rebuilds; otherwise you measure the rebuild, not steady state.
- [ ] **3 runs per configuration; report median + spread.** A single run is noise. **One variable per run.**
- [ ] Capture duration: MangoHud `log_duration=30` (seconds; match scene length); PresentMon Option B `--timed 30`. P0 extra: ~5 min thermal soak before first measured run. P0 runtime log toggle: `toggle_logging` (default `Shift_L+F2`).
- [ ] Invalid-run guardrails: FPS limiter active; governor moved (P0/P2); RAM headroom shifted materially (P0/P2); power plan changed or GPU throttling (P1) — `nvidia-smi -q -d PERFORMANCE` must show no `SW Power Cap` / `HW Slowdown`.
- [ ] GPU background logging (P1+P2, identical command): `nvidia-smi --query-gpu=timestamp,memory.used,clocks.sm,clocks.mem,power.draw,temperature.gpu,utilization.gpu --format=csv -l 1` → `gpu-<timestamp>.csv` (`memory.used` = VRAM series).
- [ ] Fleet-wide JVM: Java 21, **same JVM build on all three** (Temurin 21); identical G1 args except `-Xmx` (P0 `-Xmx5G` start/6 G max, P1 `-Xmx8G`, P2 `-Xmx6G`); **omit `-XX:+AlwaysPreTouch` on P0 and P2**.
## (b) Recording-sheet columns (one row per run)
- **P0**: `Date / session`, `Machine`, `Config ID`, `Scene`, `Run #`, `CSV file`, `p50 / p95 / p99 frametime (ms)`, `1% low FPS`, `Avg FPS`, `Peak measured frametime (ms)`, `Notes`.
- **P1**: same plus `CSV file (raw + summary)`, `VRAM peak (MB)` (headline metric), `Throttle flag` (none / SW Power Cap / HW Slowdown).
- **P2**: same plus `VRAM peak (MB)`, `Throttle flag`, `MemAvailable before / after`, `Swap delta` (must be 0).
- Example values: Machine `P0-SteamDeck-LCD` / `P1-7700K-1060` / `P2-6700K-1080Ti`; Config ID `baseline / dh-off / emf-off / …`; Scene `A-base / B-terrain / C-gui`; Run # `1 / 2 / 3`; CSV e.g. `Minecraft_2026-09-28_14-22-01.csv`.
## (c) Per-machine prep specifics
### P0 — Steam Deck (GPU/shared-RAM bound; LCD 60 Hz can only validate the 30/60 ladder rows)
- [ ] Charger plugged in; **Desktop Mode** (Gaming Mode routes through MangoApp, which cannot log); instance on **internal storage**; record panel refresh rate.
- [ ] SteamOS overlay: Quick Access `...` → Performance → **Overlay Level 0**, **Framerate Limit Off**, **TDP Off or pinned (e.g. 15 W)** — recorded, constant; check no per-game profile re-enables them; disable Decky MangoApp-routed plugins.
- [ ] Governor: leave `schedutil` as set but record it (`/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor`); pinning `performance` risks thermal throttling.
- [ ] MangoHud install (re-check every session — SteamOS updates remove pacman packages): `sudo steamos-readonly disable && sudo pacman -S --needed mangohud && sudo steamos-readonly enable`; find real paths with `pacman -Ql mangohud | grep -E 'mangohud$|libMangoHud'`. Alternative: MangoHudPy (verify it can write CSV first).
- [ ] Config from packaged example → `~/.config/MangoHud/MangoHud.conf`: `output_folder=/home/deck/mcbench/logs`, `autostart_log=1`, `log_duration=30`, `fps_limit=0`, `frametime=1`, `frame_timing=1`; percentile key/syntax: **not stated** — confirm in packaged example.
- [ ] Prism wrapper command (Custom Commands): `mangohud`; fallback `mangohud --dlsym`.
### P1 — Windows 7700K / GTX 1060 6 GB (VRAM bound; only Windows box)
- [ ] `powercfg /setactive SCHEME_MIN` (High performance); Game Bar off; Game DVR background recording off; Defender real-time exclusion for instance folder (record it); close browsers/Discord/OneDrive.
- [ ] NVIDIA per-program settings for **`javaw.exe`** (verify process name in Task Manager): Power management → **Prefer maximum performance**; Vertical sync → **Off**; Low Latency Mode, Threaded Optimization, Texture filtering → fixed + recorded; HAGS recorded + constant; disable *Fullscreen optimizations* for `javaw.exe`; same fullscreen display mode every session.
- [ ] `nvidia-smi` (PowerShell, same command as P2) + `-q -d PERFORMANCE` throttle check; VRAM logging = headline metric (peak VRAM on 6 GB).
- [ ] Tooling: **PresentMon** (no MangoHud on Windows), run **as Administrator**. Option A: Frame Time Analysis `Run.bat` → `presentmon.exe --multi_csv --no_console_stats --hotkey "F10"` (F10 starts/stops; new CSV per capture). Option B: `presentmon.exe --process_name javaw.exe --output_file ~\mcbench\logs\pm.csv --timed 30`. Emits raw CSV + stats summary CSV (avg/min/max/percentile FPS). Analysis: Frame Time Analysis (browser, reads PresentMon/CapFrameX/OCAT/MangoHud/FrameView, merges CSVs).
- [ ] Cross-OS caveat: ETW vs MangoHud don't measure identically — absolute numbers not cross-comparable; compare orderings per OS.
### P2 — Linux 6700K / GTX 1080 Ti (CPU main-thread + RAM/GC bound; ~8 GB taken by Docker)
- [ ] **Do not alter Docker or any running process** — treat load as constant; verify instead.
- [ ] Proprietary NVIDIA driver confirmed (`nvidia-smi`; `glxinfo -B | grep -E 'OpenGL renderer|OpenGL version'`); compositor vsync off; record resolution/refresh.
- [ ] `nvidia-settings -a '[gpu:0]/GPUPowerMizerMode=1'`; throttle check via `nvidia-smi -q -d PERFORMANCE`; start the shared `nvidia-smi --query-gpu=…` logging.
- [ ] **Noise floor FIRST**: same config, same scene, 3 runs, no changes — p50/p95/p99 spread = detection threshold; any delta below it is not evidence.
- [ ] RAM baseline: record `MemAvailable` (`free -h`, `/proc/meminfo`) at start and before/after each block; drift = runs not comparable; `grep -E 'pswpin|pswpout' /proc/vmstat` must not move (swap during a run invalidates p99 outright).
- [ ] Governor: `sudo cpupower frequency-set -g performance` (record; revert afterwards). Bench at stock clocks.
- [ ] MangoHud: config keys same as P0 (`output_folder=/home/<user>/mcbench/logs`, `autostart_log=1`, `log_duration=30`, `fps_limit=0`, `frametime=1`, `frame_timing=1`); percentile key: **not stated**. Prism wrapper `mangohud`, fallback `mangohud --dlsym`.
- [ ] Heap rule: `-Xmx` = min(6 GB, `MemAvailable` − 2 GB); rig **over-weights RAM/GC ladder steps** (Docker) — record as rig property, don't attribute to a mod.
## (d) LEVERS §7 corrections (every bullet, corrected fact)
- **DH default threads is 4 on the fleet, not 2** — default is `ceil(nproc × 0.5)` (`ThreadPresetConfigEventHandler` @ 3.3.2); the observed `numberOfThreads = 2` came from the owner's 4-thread machine (nproc=4). All other observed DH values match source defaults. *(Phase 1b)*
- **Puzzles Lib 21.1.64 has no perf change on NeoForge** (lever 14 is a bugfix only) — r9's lead and this file's earlier "the fix is in 21.1.64 for 1.21.1" were wrong for our loader. *(Phase 1b)*
- **r6's Shenandoah flags crash JDK 21**: `-XX:ShenandoahGCMode=iu` needs `-XX:+UnlockExperimentalVMOptions` (tested OpenJDK 21.0.12); use plain `-XX:+UseShenandoahGC`. *(Phase 1b)*
- **SSRD's distance is a COMMON config**, not a Sodium option: `config/ssrd-common.toml`, `physicsRenderDistance`, default 128, slider floor 16; the server reads it too. *(Phase 1b)*
- **r4 "remove one CTM engine" is wrong** — Chipped requires Athena and Rechiseled requires Fusion; both are hard dependencies (§5).
- **r9 "Puzzles Lib ~8 % render-thread overhead"** was measured on MC 26.2/Fabric; the fix is in 21.1.64 for 1.21.1; gain size unknown. *(superseded by the Phase 1b Puzzles bullet above — both appear in §7 as written)*
- **r3/r8's EMF #443 magnitude** is from the closed Oculus/Forge era; treat the 4× figure as directional only.
- **"Sable overrides vanilla core shaders"**: real but small — just `worldborder.*` from the nested Veil; the substantive part is Sable's runtime patching of Flywheel's internal shaders.
- **r1's Flywheel-under-shaders claim**: confirmed in `Backends.java` on the 1.21.1 branch (with any shaderpack active, Flywheel marks both GPU backends unsupported → Create renders kinetic BEs on CPU + shadow pass).