feat(bench): add datapack-driven harness v2 with boot checks
xdotool chat injection silently fails under KDE Wayland/Xwayland, so the v1 harness recorded runs whose scene commands never executed. v2 drives scenes from a generated world datapack, detects crashes, aligns MangoHud CSVs by file birth time, flags vsync/backup/low-n windows, and records full run metadata to JSONL; analyze.py applies the BENCH-MATRIX win rule.
This commit is contained in:
logan committed
1 parent
a84319bb2f
commit
e18bd3ccbb
7 files changed
+1741
No files matched your search
@@ -0,0 +1,102 @@
|
||||
# Bench harness v2
|
||||
|
||||
`bench.py` drives the OnionBench Prism instance (`~/onion-bench/prism/instances/OnionBench`)
|
||||
with **no input injection**: a generated world datapack runs the scenes and prints
|
||||
markers, the harness only reads logs and the MangoHud CSV. `run_scene.py` is the legacy
|
||||
xdotool harness, kept for reference. Stdlib only, Python 3.
|
||||
|
||||
## Loop
|
||||
|
||||
```sh
|
||||
cd analysis/client-perf/harness
|
||||
python3 bench.py boot --n 3 --variant C1 # cold launches -> results/boot.jsonl
|
||||
python3 bench.py boot --n 3 --join # ... into the world (join time)
|
||||
python3 bench.py session --variant C0 --plan s1 --reps 3 --label B0 # x3 sessions for B0
|
||||
python3 bench.py session --variant C1 --plan s1,s2,s3,s4,s5 --label B2
|
||||
python3 bench.py session --variant C1 --plan s5 --jvm J2 --heap 6144 --label B4
|
||||
python3 analyze.py --baseline C0 --noise B0 [--label B2] # markdown table + verdicts
|
||||
python3 -m unittest discover -s tests # offline tests (from this dir)
|
||||
```
|
||||
|
||||
`session --dry-run` prints the datapack it would write and changes nothing. `bench.py meta`
|
||||
prints the environment snapshot; `bench.py disarm` removes a leftover plan pack.
|
||||
|
||||
- **boot**: N launches; waits for `Game took N seconds` (or `joined the game` with
|
||||
`--join`); outcome `ok` / `crash` / `timeout`. Crash = new file in `crash-reports/` or
|
||||
`hs_err_pid*.log`, client java exited, or a log line with `Failed to create mod
|
||||
instance`, a `/FATAL]` level tag, `Cowardly refusing`, or a crash-report header.
|
||||
- **session**: applies the variant (`proposed-configs/apply_variant.py`), then pins
|
||||
`options.txt` (no onboarding/narrator, no pause on focus loss, `maxFps:260`, vsync
|
||||
off), hashes `config/` (minus `.bak`/`.tmp`/spark buffers) and `options.txt`, writes the
|
||||
plan datapack, launches once into `onionbench`, tails `latest.log` until `BENCH_DONE`,
|
||||
crash, exit or timeout, kills the client, removes the plan pack, then slices the CSV.
|
||||
One JSON row per scene x rep window goes to `~/onion-bench/results/session.jsonl`;
|
||||
every session's `latest.log` is archived under `~/onion-bench/results/logs/`.
|
||||
- Only java processes whose command line contains `instances/OnionBench` are ever
|
||||
signalled. The dedicated server (`unix_args.txt`) is only observed (running + %CPU).
|
||||
- `--jvm J0..J3` / `--heap MB` rewrite `JvmArgs` / `MaxMemAlloc` in the bench
|
||||
`instance.cfg` (first original saved as `instance.cfg.harness-orig`). Without them the
|
||||
instance is left as is; the effective `-X*` flags are read from the running client.
|
||||
|
||||
## Datapack mechanism
|
||||
|
||||
The scene functions live in the permanent pack `saves/onionbench/datapacks/onion-bench`
|
||||
(`bench:s1` ... `bench:tour_start`, source in `../bench/onion-bench`). Per session the
|
||||
harness writes `datapacks/onion-bench-autoplan/` (pack_format 48, 1.21.1 `function/`
|
||||
dirs) with a session id `pYYYYMMDDHHMMSS`:
|
||||
|
||||
- `#minecraft:load` -> `bench:autoplan/<sid>/load`: unless score `#<sid> bench_plan` is
|
||||
set, poll every second (`schedule`) until `execute if entity @a`;
|
||||
- `begin`: set the score (a `/reload` won't restart the plan), clear stale
|
||||
`storm_tick` / tour schedules, `say BENCH_PLAN <sid> begin`, wait `--settle` s;
|
||||
- chained `stepN` functions: `execute as @a[limit=1] at @s run function bench:<scene>`,
|
||||
`say BENCH_MARK warm|start|end <scene> <rep>`, `schedule` the warm-up / capture ticks;
|
||||
`say BENCH_DONE <sid>` at the end.
|
||||
|
||||
S5 is fixed by BENCH-MATRIX: `bench:s1`, 10 s, `bench:tour_start`, all 120 s captured
|
||||
(warm-up ignored). S2s = S2 + `bench:storm_on` during warm-up and capture. Plans must
|
||||
follow S1 -> S5 order (`--any-order` overrides). Function names contain the session id, so
|
||||
schedules saved into `level.dat` by an autosave can never fire in a later session. The pack
|
||||
is deleted in a `finally`, so normal play of the world is unaffected; after a hard kill
|
||||
of the harness itself run `bench.py disarm`.
|
||||
|
||||
Markers are taken from the server log line (`[Server] BENCH_MARK ...`, ms timestamps).
|
||||
The MangoHud CSV used is one created at/after launch (never a stale one); wall time =
|
||||
file birth time (`stat -c %.9W`, else the name's timestamp) + `elapsed`, and the method is
|
||||
recorded as `align`. Per window: n, p50/p95/p99/peak ms, avg fps (1000/mean frametime),
|
||||
1 % low = 1000/mean of the worst 1 % frametimes, and the legacy fps-column 1 % low.
|
||||
|
||||
Flags: `vsync_suspect` (p50 within 3 % of 60 Hz or an xrandr refresh), `low_n`
|
||||
(n < 500 or < half of duration x avg fps), `crashed`/`timeout` (window never ended),
|
||||
`session_crash` (window ended, session crashed later), `backup_in_window` (backup or
|
||||
autosave line from 5 s before start to end; warn only). `analyze.py` drops rows with
|
||||
any non-warn flag unless `--include-invalid`.
|
||||
|
||||
Metadata per row: label, machine (`proxy-9900X-7800XT`), git sha + dirty, harness sha,
|
||||
variant + config/options hashes, mods hash/count, Java version, cfg JVM args + effective
|
||||
flags, MaxMemAlloc, kernel, CPU governor, per-GPU DPM level + sclk, MangoHud's GPU
|
||||
string, MemAvailable before/after, load average, server running + %CPU, startup and
|
||||
join seconds, block-atlas size, `level.dat` hash.
|
||||
|
||||
## Known limits
|
||||
|
||||
- The desktop is **KDE Plasma on Wayland** (kwin_wayland + Xwayland at `:1`). xdotool
|
||||
only reaches Xwayland windows, `windowactivate --sync` hangs under kwin, focus-steal
|
||||
prevention can drop keystrokes, and typed chat lands in whatever has focus. Hence no
|
||||
input injection at all in v2.
|
||||
- Still manual: **S6** (JEI/GUI overlay needs a key press), **B7** shader profiles (set in
|
||||
Iris' GUI), `dh-off`-style checks needing F3 reads, `/sparkc` heap/GC reads (B1/B4).
|
||||
- MangoHud's `log_duration=1800` in `~/.local/bin/mangohud-local` caps one CSV at 30 min;
|
||||
`session` warns when a plan is longer (B5's 45-min decay run needs a split or a larger
|
||||
value).
|
||||
- The integrated server ticks the plan, so scene timing stretches if it lags; windows use
|
||||
the marker wall times, not tick counts. The marker chat line is visible for ~10 s at
|
||||
the start of each capture (identical across variants).
|
||||
- **FTB Backups 3** backs up the world right after join (its index
|
||||
`ftbbackups3/backups.json` doesn't parse, so it treats the world as never backed up)
|
||||
and every `backup_timer` = 120 min. For the bench instance only, set `auto: false` in
|
||||
`saves/onionbench/serverconfig/ftbbackups3-server.snbt` (copy of
|
||||
`config/ftbbackups3-server.snbt`; the world override wins and is never touched by pack
|
||||
updates). Not done by the harness. Never edit the repo's `config/` for this.
|
||||
- The dedicated server in tmux `onion-mc` keeps running during benches and was observed
|
||||
using ~78 % of a core while idle; it is recorded per row as a noise covariate.
|
||||
@@ -0,0 +1,155 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Summarise bench.py session rows and apply the BENCH-MATRIX win rule.
|
||||
|
||||
analyze.py [--file ~/onion-bench/results/session.jsonl] [--label B2]
|
||||
[--baseline C0 --noise B0] [--include-invalid] [--exclude-backup]
|
||||
|
||||
Groups windows by (label, variant, scene). Each session contributes its median p99 and
|
||||
median 1 % low (frametime-based); a group's value is the median of its session medians
|
||||
and its spread is max - min of those. With --baseline/--noise, every non-baseline group
|
||||
is compared with the baseline variant on the same scene (same label if present, else
|
||||
all labels pooled): win = median p99 improves by more than the noise label's p99 spread
|
||||
AND 1 % low does not worsen by more than its 1 % low spread; loss = p99 worsens beyond
|
||||
the spread or 1 % low drops beyond it; otherwise tie (ties keep upstream defaults).
|
||||
Rows flagged crashed / timeout / no_data / no_markers / low_n / vsync_suspect are
|
||||
excluded unless --include-invalid. Stdlib only.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import statistics
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
INVALID = {"crashed", "timeout", "no_data", "no_markers", "low_n", "vsync_suspect"}
|
||||
DEFAULT_FILE = Path.home() / "onion-bench" / "results" / "session.jsonl"
|
||||
|
||||
|
||||
def load_rows(path):
|
||||
rows = []
|
||||
for line in Path(path).read_text().splitlines():
|
||||
line = line.strip()
|
||||
if line:
|
||||
rows.append(json.loads(line))
|
||||
return [r for r in rows if r.get("kind", "window") == "window"]
|
||||
|
||||
|
||||
def is_valid(row, exclude_backup=False):
|
||||
flags = set(row.get("flags") or [])
|
||||
if not row.get("metrics") or flags & INVALID:
|
||||
return False
|
||||
return not (exclude_backup and "backup_in_window" in flags)
|
||||
|
||||
|
||||
def summarise(rows):
|
||||
"""{(label, variant, scene): stats} from valid window rows."""
|
||||
by_session = {}
|
||||
for r in rows:
|
||||
key = (r.get("label"), r["variant"], r["scene"])
|
||||
by_session.setdefault(key, {}).setdefault(r["session"], []).append(r["metrics"])
|
||||
out = {}
|
||||
for key, sessions in by_session.items():
|
||||
p99s = [statistics.median(m["p99_ms"] for m in ms) for ms in sessions.values()]
|
||||
lows = [statistics.median(m["one_low_fps"] for m in ms) for ms in sessions.values()]
|
||||
fps = [statistics.median(m["avg_fps"] for m in ms) for ms in sessions.values()]
|
||||
out[key] = {
|
||||
"sessions": len(sessions),
|
||||
"windows": sum(len(ms) for ms in sessions.values()),
|
||||
"p99": statistics.median(p99s), "p99_spread": max(p99s) - min(p99s),
|
||||
"low": statistics.median(lows), "low_spread": max(lows) - min(lows),
|
||||
"fps": statistics.median(fps),
|
||||
}
|
||||
return out
|
||||
|
||||
|
||||
def noise_spread(summary, noise_label):
|
||||
"""(p99 spread, 1 % low spread) of the noise block, worst case across its groups."""
|
||||
groups = [v for (lab, _, _), v in summary.items() if lab == noise_label]
|
||||
if not groups:
|
||||
return None
|
||||
return max(g["p99_spread"] for g in groups), max(g["low_spread"] for g in groups)
|
||||
|
||||
|
||||
def verdict(base, cand, spread_p99, spread_low):
|
||||
gain = base["p99"] - cand["p99"] # > 0: candidate's p99 is lower (better)
|
||||
low_drop = base["low"] - cand["low"] # > 0: candidate's 1 % low is worse
|
||||
if gain > spread_p99 and low_drop <= spread_low:
|
||||
return "win"
|
||||
if gain < -spread_p99 or low_drop > spread_low:
|
||||
return "loss"
|
||||
return "tie"
|
||||
|
||||
|
||||
def baseline_for(summary, rows, label, scene, baseline):
|
||||
if (label, baseline, scene) in summary:
|
||||
return summary[(label, baseline, scene)], label
|
||||
pooled = [r for r in rows if r["variant"] == baseline and r["scene"] == scene]
|
||||
if not pooled:
|
||||
return None, None
|
||||
s = summarise([dict(r, label="*") for r in pooled])
|
||||
return s.get(("*", baseline, scene)), "*"
|
||||
|
||||
|
||||
def table(summary, rows, baseline=None, noise=None):
|
||||
spreads = noise_spread(summary, noise) if noise else None
|
||||
hdr = ["label", "variant", "scene", "sess", "win#", "p99 ms", "p99 spr", "1%low",
|
||||
"1%low spr", "avg fps"]
|
||||
if baseline:
|
||||
hdr += ["vs", "Δp99 ms", "Δ1%low", "verdict"]
|
||||
out = ["| " + " | ".join(hdr) + " |", "|" + "---|" * len(hdr)]
|
||||
for key in sorted(summary, key=lambda k: (str(k[0]), k[2], k[1])):
|
||||
lab, var, scene = key
|
||||
s = summary[key]
|
||||
cells = [str(lab), var, scene, str(s["sessions"]), str(s["windows"]),
|
||||
f"{s['p99']:.2f}", f"{s['p99_spread']:.2f}", f"{s['low']:.1f}",
|
||||
f"{s['low_spread']:.1f}", f"{s['fps']:.0f}"]
|
||||
if baseline:
|
||||
if var == baseline:
|
||||
cells += ["—", "", "", "baseline"]
|
||||
else:
|
||||
b, blab = baseline_for(summary, rows, lab, scene, baseline)
|
||||
if b is None:
|
||||
cells += ["—", "", "", "no baseline"]
|
||||
elif spreads is None:
|
||||
cells += [f"{baseline}@{blab}", f"{b['p99'] - s['p99']:+.2f}",
|
||||
f"{s['low'] - b['low']:+.1f}", "no noise data"]
|
||||
else:
|
||||
cells += [f"{baseline}@{blab}", f"{b['p99'] - s['p99']:+.2f}",
|
||||
f"{s['low'] - b['low']:+.1f}", verdict(b, s, *spreads)]
|
||||
out.append("| " + " | ".join(cells) + " |")
|
||||
if baseline and noise:
|
||||
out.append("")
|
||||
out.append(f"noise ({noise}): " + (
|
||||
f"p99 spread {spreads[0]:.2f} ms, 1%low spread {spreads[1]:.1f} fps"
|
||||
if spreads else "no rows - verdicts unavailable"))
|
||||
return "\n".join(out)
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
|
||||
ap.add_argument("--file", default=str(DEFAULT_FILE))
|
||||
ap.add_argument("--label", action="append", help="only these labels (repeatable); "
|
||||
"the noise label is always kept")
|
||||
ap.add_argument("--baseline", help="baseline variant, e.g. C0")
|
||||
ap.add_argument("--noise", help="label whose spread is the win threshold, e.g. B0")
|
||||
ap.add_argument("--include-invalid", action="store_true")
|
||||
ap.add_argument("--exclude-backup", action="store_true")
|
||||
a = ap.parse_args(argv)
|
||||
rows = load_rows(a.file)
|
||||
total = len(rows)
|
||||
if not a.include_invalid:
|
||||
rows = [r for r in rows if is_valid(r, a.exclude_backup)]
|
||||
warn = sum(1 for r in rows if "backup_in_window" in (r.get("flags") or []))
|
||||
shown = rows
|
||||
if a.label:
|
||||
keep = set(a.label) | ({a.noise} if a.noise else set())
|
||||
shown = [r for r in rows if r.get("label") in keep]
|
||||
if not shown:
|
||||
print("no valid rows", file=sys.stderr)
|
||||
return 1
|
||||
print(table(summarise(shown), rows, a.baseline, a.noise))
|
||||
print(f"\n{len(rows)}/{total} windows valid; {warn} with backup_in_window (warn only)")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,4 @@
|
||||
[14Oct2026 22:21:53.972] [modloading-worker-0/ERROR] [net.neoforged.fml.javafmlmod.FMLModContainer/LOADING]: Failed to create mod instance. ModID: midnightlib, class eu.midnightdust.core.MidnightLib
|
||||
[14Oct2026 22:21:57.100] [Render thread/FATAL] [net.minecraft.client.Minecraft/]: Unreported exception thrown!
|
||||
[14Oct2026 22:21:57.200] [Render thread/ERROR] [net.neoforged.fml.ModLoader/]: Cowardly refusing to send event net.neoforged.neoforge.client.event.RenderFrameEvent to a broken mod state
|
||||
[14Oct2026 22:21:57.300] [Render thread/INFO] [mixin/]: harmless line mentioning FATAL in text only
|
||||
@@ -0,0 +1,11 @@
|
||||
[15Oct2026 15:25:44.104] [Render thread/INFO] [net.minecraft.client.renderer.texture.TextureAtlas/]: Created: 4096x4096x2 minecraft:textures/atlas/blocks.png-atlas
|
||||
[15Oct2026 15:25:52.383] [Render thread/WARN] [ModernFix/]: Game took 28.366 seconds to start
|
||||
[15Oct2026 15:25:53.147] [Server thread/INFO] [net.minecraft.server.MinecraftServer/]: Onion4Dinner joined the game
|
||||
[15Oct2026 15:25:59.974] [Server thread/INFO] [net.minecraft.server.MinecraftServer/]: Server Backup started!
|
||||
[15Oct2026 15:26:03.000] [Server thread/INFO] [net.minecraft.server.MinecraftServer/]: [Server] BENCH_PLAN p20261015152526 begin
|
||||
[15Oct2026 15:26:13.000] [Server thread/INFO] [net.minecraft.server.MinecraftServer/]: [Server] BENCH_MARK warm s1 1
|
||||
[15Oct2026 15:26:13.050] [Render thread/INFO] [net.minecraft.client.gui.components.ChatComponent/]: [System] [CHAT] [Server] BENCH_MARK warm s1 1
|
||||
[15Oct2026 15:26:28.000] [Server thread/INFO] [net.minecraft.server.MinecraftServer/]: [Server] BENCH_MARK start s1 1
|
||||
[15Oct2026 15:26:28.040] [Render thread/INFO] [net.minecraft.client.gui.components.ChatComponent/]: [System] [CHAT] [Server] BENCH_MARK start s1 1
|
||||
[15Oct2026 15:26:58.000] [Server thread/INFO] [net.minecraft.server.MinecraftServer/]: [Server] BENCH_MARK end s1 1
|
||||
[15Oct2026 15:26:58.000] [Server thread/INFO] [net.minecraft.server.MinecraftServer/]: [Server] BENCH_DONE p20261015152526
|
||||
+204
@@ -0,0 +1,204 @@
|
||||
os,cpu,gpu,ram,kernel,driver,cpuscheduler
|
||||
Manjaro Linux,AMD Ryzen 9 9900X 12-Core Processor,AMD Radeon RX 7800 XT ,96396388,7.1.8-1-MANJARO,,performance
|
||||
fps,frametime,cpu_load,cpu_power,gpu_load,cpu_temp,gpu_temp,gpu_core_clock,gpu_mem_clock,gpu_vram_used,gpu_power,ram_used,swap_used,process_rss,cpu_mhz,elapsed
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,100000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,200000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,300000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,400000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,500000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,600000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,700000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,800000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,900000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,1000000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,1100000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,1200000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,1300000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,1400000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,1500000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,1600000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,1700000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,1800000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,1900000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,2000000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,2100000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,2200000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,2300000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,2400000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,2500000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,2600000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,2700000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,2800000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,2900000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,3000000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,3100000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,3200000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,3300000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,3400000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,3500000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,3600000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,3700000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,3800000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,3900000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,4000000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,4100000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,4200000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,4300000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,4400000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,4500000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,4600000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,4700000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,4800000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,4900000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,5000000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,5100000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,5200000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,5300000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,5400000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,5500000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,5600000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,5700000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,5800000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,5900000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,6000000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,6100000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,6200000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,6300000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,6400000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,6500000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,6600000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,6700000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,6800000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,6900000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,7000000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,7100000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,7200000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,7300000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,7400000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,7500000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,7600000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,7700000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,7800000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,7900000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,8000000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,8100000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,8200000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,8300000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,8400000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,8500000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,8600000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,8700000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,8800000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,8900000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,9000000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,9100000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,9200000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,9300000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,9400000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,9500000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,9600000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,9700000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,9800000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,9900000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,10000000000
|
||||
50.0000,20.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,10100000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,10200000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,10300000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,10400000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,10500000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,10600000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,10700000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,10800000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,10900000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,11000000000
|
||||
50.0000,20.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,11100000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,11200000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,11300000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,11400000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,11500000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,11600000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,11700000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,11800000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,11900000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,12000000000
|
||||
50.0000,20.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,12100000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,12200000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,12300000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,12400000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,12500000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,12600000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,12700000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,12800000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,12900000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,13000000000
|
||||
50.0000,20.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,13100000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,13200000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,13300000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,13400000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,13500000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,13600000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,13700000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,13800000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,13900000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,14000000000
|
||||
50.0000,20.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,14100000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,14200000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,14300000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,14400000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,14500000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,14600000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,14700000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,14800000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,14900000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,15000000000
|
||||
50.0000,20.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,15100000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,15200000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,15300000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,15400000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,15500000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,15600000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,15700000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,15800000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,15900000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,16000000000
|
||||
50.0000,20.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,16100000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,16200000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,16300000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,16400000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,16500000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,16600000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,16700000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,16800000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,16900000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,17000000000
|
||||
50.0000,20.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,17100000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,17200000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,17300000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,17400000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,17500000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,17600000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,17700000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,17800000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,17900000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,18000000000
|
||||
50.0000,20.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,18100000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,18200000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,18300000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,18400000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,18500000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,18600000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,18700000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,18800000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,18900000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,19000000000
|
||||
50.0000,20.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,19100000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,19200000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,19300000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,19400000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,19500000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,19600000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,19700000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,19800000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,19900000000
|
||||
250.0000,4.0000,0,0,0,0,0,0,0,0,0,0,0,0,0,20000000000
|
||||
truncated,row
|
||||
|
@@ -0,0 +1,241 @@
|
||||
"""Offline tests for bench.py / analyze.py (no game, no display).
|
||||
|
||||
python3 -m unittest discover -s analysis/client-perf/harness/tests
|
||||
"""
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import shutil
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
sys.path.insert(0, str(HERE.parent))
|
||||
|
||||
import analyze # noqa: E402
|
||||
import bench # noqa: E402
|
||||
|
||||
FIX = HERE / "fixtures"
|
||||
CSV = FIX / "mangohud_java_2026-10-15_15-25-26.csv"
|
||||
|
||||
|
||||
def lines(name):
|
||||
return (FIX / name).read_text().splitlines()
|
||||
|
||||
|
||||
class CsvParsing(unittest.TestCase):
|
||||
def test_header_and_rows(self):
|
||||
meta, rows = bench.parse_mangohud_csv(CSV)
|
||||
self.assertIn("RX 7800 XT", meta["gpu"])
|
||||
self.assertEqual(len(rows), 200) # malformed trailing row skipped
|
||||
self.assertAlmostEqual(rows[0][0], 0.1) # elapsed ns -> s
|
||||
self.assertEqual(rows[0][1], 4.0)
|
||||
|
||||
def test_name_time(self):
|
||||
import time
|
||||
t = bench.csv_name_time("java_2026-10-15_15-25-26.csv")
|
||||
self.assertEqual(time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(t)),
|
||||
"2026-10-15 15:25:26")
|
||||
self.assertIsNone(bench.csv_name_time("weird.csv"))
|
||||
|
||||
def test_file_birth_method(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
p = Path(d) / CSV.name
|
||||
shutil.copyfile(CSV, p)
|
||||
t, method = bench.file_birth(p)
|
||||
self.assertIn(method, {"st_birthtime", "stat-W", "filename", "mtime"})
|
||||
self.assertGreater(t, 0)
|
||||
|
||||
|
||||
class Math(unittest.TestCase):
|
||||
def test_pct(self):
|
||||
self.assertEqual(bench.pct([1, 2, 3, 4, 5], 0.5), 3)
|
||||
self.assertAlmostEqual(bench.pct(list(range(1, 101)), 0.99), 99.01)
|
||||
self.assertNotEqual(bench.pct([], 0.5), bench.pct([], 0.5)) # nan
|
||||
|
||||
def test_one_low_from_frametimes(self):
|
||||
# 200 frames: 198 x 5 ms, 2 x 25 ms -> worst 1 % (2 frames) mean 25 ms -> 40 fps
|
||||
rows = [(i * 0.005, 25.0 if i in (50, 150) else 5.0, 200.0) for i in range(200)]
|
||||
m = bench.window_metrics(rows, 1000.0, 1000.0, 1001.0)
|
||||
self.assertEqual(m["n"], 200)
|
||||
self.assertEqual(m["one_low_fps"], 40.0)
|
||||
self.assertEqual(m["p50_ms"], 5.0)
|
||||
self.assertEqual(m["peak_ms"], 25.0)
|
||||
self.assertAlmostEqual(m["avg_fps"], 1000 / (1040 / 200), places=1)
|
||||
self.assertEqual(m["one_low_fps_legacy"], 200.0) # fps column was flat
|
||||
|
||||
def test_window_slicing_alignment(self):
|
||||
_, rows = bench.parse_mangohud_csv(CSV)
|
||||
origin = 5000.0
|
||||
m = bench.window_metrics(rows, origin, origin + 10.0, origin + 20.0)
|
||||
self.assertEqual(m["n"], 101)
|
||||
self.assertEqual(m["p50_ms"], 4.0)
|
||||
self.assertEqual(m["p99_ms"], 20.0)
|
||||
self.assertEqual(m["one_low_fps"], 50.0)
|
||||
self.assertEqual(m["one_low_fps_legacy"], 50.0)
|
||||
self.assertIsNone(bench.window_metrics(rows, origin, origin + 30, origin + 40))
|
||||
# first 10 s contain no 20 ms frames
|
||||
early = bench.window_metrics(rows, origin, origin, origin + 9.9)
|
||||
self.assertEqual(early["p99_ms"], 4.0)
|
||||
|
||||
|
||||
class Markers(unittest.TestCase):
|
||||
def test_parse(self):
|
||||
marks, done = bench.parse_markers(lines("latest.log"))
|
||||
st, en = marks[("start", "s1", 1)], marks[("end", "s1", 1)]
|
||||
self.assertAlmostEqual(en - st, 30.0, places=3)
|
||||
self.assertAlmostEqual(marks[("warm", "s1", 1)] % 1, 0.0, places=3) # server line, not CHAT
|
||||
self.assertAlmostEqual(done, en)
|
||||
|
||||
def test_events_and_alignment(self):
|
||||
ln = lines("latest.log")
|
||||
ev = bench.scan_events(ln)
|
||||
self.assertEqual(ev["game_took_s"], 28.366)
|
||||
self.assertEqual(ev["atlas"], "4096x4096x2")
|
||||
self.assertIsNone(ev["crash"])
|
||||
self.assertAlmostEqual(ev["joined_t"] - ev["title_t"], 0.764, places=3)
|
||||
# CSV born 10 s before the start marker -> window covers elapsed 10..40 s,
|
||||
# CSV ends at 20 s -> 101 frames -> low_n
|
||||
marks, _ = bench.parse_markers(ln)
|
||||
ts, te = marks[("start", "s1", 1)], marks[("end", "s1", 1)]
|
||||
_, rows = bench.parse_mangohud_csv(CSV)
|
||||
m = bench.window_metrics(rows, ts - 10.0, ts, te)
|
||||
self.assertEqual(m["n"], 101)
|
||||
self.assertIn("low_n", bench.validity_flags(m, te - ts, [], bench.backup_times(ln), ts, te))
|
||||
|
||||
def test_backup_flag(self):
|
||||
ln = lines("latest.log")
|
||||
bk = bench.backup_times(ln)
|
||||
self.assertEqual(len(bk), 1)
|
||||
m = {"n": 6000, "p50_ms": 4.0, "avg_fps": 200.0}
|
||||
self.assertEqual(bench.validity_flags(m, 30, [], bk, bk[0] + 30, bk[0] + 60), [])
|
||||
self.assertIn("backup_in_window", bench.validity_flags(m, 30, [], bk, bk[0] - 1, bk[0] + 29))
|
||||
self.assertIn("backup_in_window", bench.validity_flags(m, 30, [], bk, bk[0] + 3, bk[0] + 33))
|
||||
|
||||
def test_vsync_flag(self):
|
||||
m = {"n": 1800, "p50_ms": 16.7, "avg_fps": 59.8}
|
||||
self.assertIn("vsync_suspect", bench.validity_flags(m, 30))
|
||||
m144 = {"n": 4320, "p50_ms": 6.95, "avg_fps": 144}
|
||||
self.assertIn("vsync_suspect", bench.validity_flags(m144, 30, [144.0]))
|
||||
self.assertNotIn("vsync_suspect", bench.validity_flags(m144, 30, [60.0]))
|
||||
self.assertEqual(bench.validity_flags(None, 30), ["no_data"])
|
||||
|
||||
|
||||
class Crash(unittest.TestCase):
|
||||
def test_classify(self):
|
||||
got = [bench.classify_line(l) for l in lines("crash.log")]
|
||||
self.assertEqual(got, ["mod_instance", "fatal", "broken_mod_state", None])
|
||||
self.assertIsNone(bench.classify_line(lines("latest.log")[3])) # backup line
|
||||
self.assertEqual(bench.scan_events(lines("crash.log"))["crash"], "mod_instance")
|
||||
|
||||
|
||||
class Datapack(unittest.TestCase):
|
||||
def test_generate(self):
|
||||
files = bench.generate_datapack("p1", ["s1", "s2s", "s5"], 2, 15, 30, 10)
|
||||
tag = json.loads(files["data/minecraft/tags/function/load.json"])
|
||||
self.assertEqual(tag["values"], ["bench:autoplan/p1/load"])
|
||||
self.assertEqual(json.loads(files["pack.mcmeta"])["pack"]["pack_format"], 48)
|
||||
load = files["data/bench/function/autoplan/p1/load.mcfunction"]
|
||||
self.assertIn("unless score #p1 bench_plan matches 1..", load)
|
||||
steps = sorted((k for k in files if "/step" in k),
|
||||
key=lambda k: int(k.rsplit("step", 1)[1].split(".")[0]))
|
||||
body = "".join(files[k] for k in steps)
|
||||
for kind in ("warm", "start", "end"):
|
||||
for sc in ("s1", "s2s", "s5"):
|
||||
for rep in (1, 2):
|
||||
self.assertEqual(body.count(f"say BENCH_MARK {kind} {sc} {rep}\n"), 1)
|
||||
self.assertIn("function bench:tour_start", body)
|
||||
self.assertIn("function bench:storm_on", body)
|
||||
self.assertIn("function bench:storm_off", body)
|
||||
self.assertIn("schedule function bench:autoplan/p1/step", body)
|
||||
self.assertNotIn(" 0t", body)
|
||||
self.assertTrue(body.rstrip().endswith("say BENCH_DONE p1"))
|
||||
|
||||
def test_timeline_ticks(self):
|
||||
tl = bench.build_timeline(["s1", "s5"], 1, 15, 30, "x")
|
||||
delays = [d for _, d in tl]
|
||||
self.assertEqual(delays, [300, 600, 200, 2400, 0])
|
||||
self.assertEqual(tl[1][0], ["say BENCH_MARK start s1 1"])
|
||||
self.assertIn("say BENCH_MARK end s1 1", tl[2][0]) # merged into next scene's step
|
||||
self.assertEqual(bench.plan_seconds(["s1", "s5"], 1, 15, 30), 45 + 130)
|
||||
|
||||
def test_order_rule(self):
|
||||
class A:
|
||||
plan, any_order = "s2,s1", False
|
||||
with self.assertRaises(SystemExit):
|
||||
bench.session_plan(A)
|
||||
A.any_order = True
|
||||
self.assertEqual(bench.session_plan(A), ["s2", "s1"])
|
||||
A.plan, A.any_order = "s1,s6", False
|
||||
with self.assertRaises(SystemExit):
|
||||
bench.session_plan(A)
|
||||
|
||||
def test_pins(self):
|
||||
out = bench.merge_pins("version:3955\nmaxFps:120\nenableVsync:true\nfov:0.5\n")
|
||||
self.assertIn("maxFps:260", out)
|
||||
self.assertIn("enableVsync:false", out)
|
||||
self.assertIn("fov:0.5", out)
|
||||
self.assertIn("pauseOnLostFocus:false", out)
|
||||
self.assertEqual(out.count("maxFps"), 1)
|
||||
|
||||
|
||||
def row(label, variant, session, p99, low, scene="s1", flags=()):
|
||||
return {"kind": "window", "label": label, "variant": variant, "scene": scene,
|
||||
"session": session, "flags": list(flags),
|
||||
"metrics": {"p99_ms": p99, "one_low_fps": low, "avg_fps": 200.0}}
|
||||
|
||||
|
||||
class Analyze(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.rows = [
|
||||
row("B0", "C0", "a", 10.0, 100), row("B0", "C0", "b", 10.2, 98),
|
||||
row("B0", "C0", "c", 10.4, 99),
|
||||
row("B2", "C0", "d", 10.2, 99),
|
||||
row("B2", "WIN", "e", 9.0, 100), row("B2", "TIE", "f", 10.0, 98),
|
||||
row("B2", "LOSS", "g", 9.0, 90),
|
||||
row("B2", "BAD", "h", 1.0, 999, flags=["crashed"]),
|
||||
]
|
||||
|
||||
def test_summary_spread(self):
|
||||
s = analyze.summarise(self.rows[:3])
|
||||
g = s[("B0", "C0", "s1")]
|
||||
self.assertEqual(g["sessions"], 3)
|
||||
self.assertAlmostEqual(g["p99"], 10.2)
|
||||
self.assertAlmostEqual(g["p99_spread"], 0.4)
|
||||
self.assertAlmostEqual(g["low_spread"], 2)
|
||||
|
||||
def test_verdicts(self):
|
||||
valid = [r for r in self.rows if analyze.is_valid(r)]
|
||||
self.assertEqual(len(valid), 7)
|
||||
s = analyze.summarise(valid)
|
||||
sp = analyze.noise_spread(s, "B0")
|
||||
base = s[("B2", "C0", "s1")]
|
||||
self.assertEqual(analyze.verdict(base, s[("B2", "WIN", "s1")], *sp), "win")
|
||||
self.assertEqual(analyze.verdict(base, s[("B2", "TIE", "s1")], *sp), "tie")
|
||||
self.assertEqual(analyze.verdict(base, s[("B2", "LOSS", "s1")], *sp), "loss")
|
||||
t = analyze.table(s, valid, "C0", "B0")
|
||||
self.assertIn("| B2 | WIN | s1 |", t)
|
||||
self.assertIn("| win |", t)
|
||||
self.assertNotIn("BAD", t)
|
||||
|
||||
def test_pooled_baseline_and_cli(self):
|
||||
rows = [r for r in self.rows if not (r["label"] == "B2" and r["variant"] == "C0")]
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
p = Path(d) / "s.jsonl"
|
||||
p.write_text("".join(json.dumps(r) + "\n" for r in rows))
|
||||
loaded = analyze.load_rows(p)
|
||||
s = analyze.summarise([r for r in loaded if analyze.is_valid(r)])
|
||||
b, lab = analyze.baseline_for(s, loaded, "B2", "s1", "C0")
|
||||
self.assertEqual(lab, "*")
|
||||
self.assertAlmostEqual(b["p99"], 10.2)
|
||||
buf = io.StringIO()
|
||||
with contextlib.redirect_stdout(buf):
|
||||
rc = analyze.main(["--file", str(p), "--baseline", "C0", "--noise", "B0"])
|
||||
self.assertEqual(rc, 0)
|
||||
self.assertIn("| B2 | WIN | s1 |", buf.getvalue())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in new issue
Block a user