logan
60e10b35a3
apply_jvm wrote instance.cfg and never restored it, so B5-B12a in run 1 ran on B4's last set (Shenandoah, 6 GB). Within-block verdicts hold (same JVM on both sides); B7's no-shader baseline is re-run in run 2. Also: B14 spawn-count automation (b14/: fresh seeded world per run, offline-located biome sites, datapack entity counts, journaled config variants) and RESULTS-proxy.md with run-1 verdicts.
103 lines
8.0 KiB
Markdown
103 lines
8.0 KiB
Markdown
# Bench results — proxy machine, run 1 (B0–B12a)
|
||
|
||
**Proxy data only.** One machine (Ryzen 9 9900X / RX 7800 XT, Manjaro), not the fleet (P0 Deck,
|
||
P1 7700K+1060, P2 6700K+1080 Ti). Run 1: 2026-10-16 22:25 → ~2026-10-17 05:00 EDT, 56 sessions
|
||
(57 launches incl. one auto-retry), 306/309 capture windows valid. Source:
|
||
`~/onion-bench/results/session.jsonl` (`ts >= 2026-10-16T22:24`), verdicts from
|
||
`harness/analyze.py --baseline C1 --noise B0 --since 2026-10-16T22:24`.
|
||
Run 2 (B4h, B7 shader rows, B12b) is in progress; B13/B14 are manual and not done.
|
||
|
||
## Method
|
||
|
||
- **Harness**: `harness/bench.py` + `matrix.py`. Every session is one cold launch at
|
||
1920×1080 in headless gamescope, world restored from `bench-world-v1*.zip` (DH LODs
|
||
pre-generated, `generatorPlan=DISABLED`), scenes driven by an in-world datapack, spectator
|
||
camera, FTB Backups off, dedicated server stopped.
|
||
- **Captures**: per scene 15 s warm-up + 30 s capture (S5 tour 120 s, no warm-up) × 3 reps,
|
||
45 s settle after join. Frametimes from MangoHud; metrics = p50/p99 frametime, 1 % low
|
||
(frametime-based), avg fps.
|
||
- **Win rule (BENCH-MATRIX §4)**: win = median p99 improves by more than the B0 p99 spread
|
||
AND 1 % low doesn't drop by more than the B0 1 %-low spread; loss = worse beyond either;
|
||
else tie (ties keep the current/upstream value).
|
||
- **Noise floor (B0, C0 × 3 sessions on S1)**: p99 spread **0.40 ms**, 1 %-low spread
|
||
**6.6 fps** (p99 9.64 ms, 1 % low 88.3 fps).
|
||
- **Baselines**: each variant is compared with the C1 session(s) of its own block
|
||
(`C1@B3`, `C1@B11.baseline`, …), i.e. same JVM and same time window.
|
||
|
||
## Highlights
|
||
|
||
- **The shipped tuning (C1) beats upstream defaults (C0) on every scene.** C0 (B1) vs C1
|
||
(B2), same JVM (Prism default, 8 GB):
|
||
|
||
| Scene | p99 C0 → C1 (ms) | 1 % low C0 → C1 (fps) | avg fps C0 → C1 |
|
||
|---|---|---|---|
|
||
| S1 vista | 9.33 → 8.23 | 93.0 → 104.9 | 218 → 272 |
|
||
| S2 Create floor | 3.79 → 3.22 | 218.7 → 259.8 | 694 → 975 |
|
||
| S3 mob pen | 7.43 → 4.46 | 110.1 → 186.0 | 255 → 395 |
|
||
| S4 airships | 2.93 → 2.44 | 298.0 → 341.3 | 789 → 1073 |
|
||
| S5 tour | 3.47 → 2.95 | 194.2 → 223.8 | 788 → 1017 |
|
||
|
||
Startup ("Game took"): C0 31.38 s → C1 25.36 s. Atlas: C0 4096×4096×4 → C1 4096×4096×2.
|
||
- **Distant Horizons is the biggest single render cost.** Turning DH off (B3.dh-off) gives
|
||
S1 p99 7.99 → 5.38 ms, 1 % low 105.5 → 144.0; S5 3.09 → 2.34 ms, 211.2 → 282.4.
|
||
DH's LOW quality preset at C1's radius 128 (`dh-qlow`) is a **win** on S1 (−0.96 ms,
|
||
+9.5 fps). SSAO costs 1.25 ms p99 on S1.
|
||
- **Fresh Animations is expensive.** S3 p99 4.32 → 8.65 ms (FA) / 10.20 ms (FA +
|
||
Extensions); 1 % low 184.6 → 104.9 / 89.0. It is opt-in (not enabled by default) — keep
|
||
it that way and document the cost.
|
||
- **The added performance mods earn their place.** Removing BadOptimizations, Particle Core
|
||
or Sodium Extra each loses (Particle Core / Sodium Extra: S3 1 % low −24 / −25 fps).
|
||
- **ModernFix `dynamic_resources` (B6)**: startup 7.6 s faster (26.32 vs 33.92 s median) and
|
||
better frametimes with it on. Ship is gated only on the manual missing-texture check
|
||
(see B6 row).
|
||
|
||
## Per-block verdicts
|
||
|
||
Numbers are p99 ms / 1 % low fps; Δ is vs the block's C1 baseline (+ = better p99).
|
||
|
||
| Block | Change tested | Result | Verdict | Ship action |
|
||
|---|---|---|---|---|
|
||
| B0 | noise floor, C0 × 3 on S1 | p99 9.64 (spread 0.40), 1 % low 88.3 (spread 6.6) | — | thresholds for all blocks |
|
||
| B1 | C0 baseline, S1–S5 | see highlights table; startup 31.38 s | reference | — |
|
||
| B2 | C1 vs C0 | C1 wins S1–S5 (table above) | **win** | C1 stays shipped; no bisect needed |
|
||
| B2 | C1-deck vs C1 | S1 7.08/115.0 vs 8.23/104.9 (win); S2/S4/S5 tie; S3 4.84/167.5 vs 4.46/186.0 (loss) | mixed | keep Deck tier as is (S3 loss: check `entityShadows`/particles on fleet P0) |
|
||
| B2 | C1-high vs C1 | S1 tie; S2–S5 losses (e.g. S4 2.66/311.6 vs 2.44/341.3) | expected cost | keep High as opt-in quality tier |
|
||
| B3 | DH radius 64 | S1 7.46/111.1 vs 7.99/105.5 (win); S5 tie | win S1 | already the Deck value; no change to C1 |
|
||
| B3 | DH radius 192 / 256 | S1 tie (7.69/105.5, 8.07/101.5); S5 tie | tie | **decision**: on this GPU 256 (upstream) ties 128; fleet floors decide — keep 128 until fleet data |
|
||
| B3 | DH LOW quality preset @ r128 (`dh-qlow`) | S1 7.03/115.0 vs 7.99/105.5 (+0.96, +9.5) | **win** (S1 only) | candidate for DEFAULT after an S5 run + visual check |
|
||
| B3 | DH SSAO on | S1 9.24/95.3 (−1.25, −10.2) | loss | confirms C1 SSAO off |
|
||
| B3 | DH threads 4 | S1 tie; S5 3.32/197.4 vs 3.09/211.2 (loss) | loss | threads stay 2 |
|
||
| B3 | DH off (reference) | S1 5.38/144.0; S5 2.34/282.4 | — | cost reference only |
|
||
| B4 | GC J0/J1/J2/J3 @ 6 GB, S2+S5 | S5: J0 3.07/226.8, J1 3.11/228.1 (tie), J2 3.40/216.6 (loss), J3 3.75/173.4 (loss); S2 all tie | **J0** | keep J0; heap ladder = B4h (run 2) |
|
||
| B5 | Sable ships 0/1/4 (proxy) | S4 3.00/280.6, 3.12/261.6, 2.97/273.2 | — | ships nearly free on this GPU |
|
||
| B5 | SSRD 32/64/128, sable-noshade (4 ships) | 3.02/265.0, 3.03/264.7, 3.11/261.9, 3.04/263.0 vs pooled C1 (−8 to −11 fps 1 % low) | loss/noise | keep C1 (SSRD 16, shading on); decay run deferred (fleet P0/P1) |
|
||
| B6 | ModernFix dynamic_resources off vs on (C1) | S1 9.11/83.8 vs 8.48/98.6; S2 3.29/236.1 vs 3.18/264.0; startup 33.92 vs 26.32 s | dynres on **wins** | ship `modernfix-mixins.properties` **after** the manual missing-texture check |
|
||
| B7 | shaders ×4 profiles | no-shader baseline only (S1 8.24/105.3, S2 3.21/266.9, S4 2.53/326.0) | pending run 2 | — |
|
||
| B8 | Fresh Animations (+Ext) on S3 | 8.65/104.9 (FA), 10.20/89.0 (FA+Ext) vs 4.32/184.6 | large cost | keep opt-in; document in PLAYER-TUNING |
|
||
| B8 | EMF animation LOD 20 / 8 (with FA) | vs C1-fa (LOD 12) 8.65/104.9: LOD 20 8.91/101.8 (tie), LOD 8 8.16/108.0 (+0.49, +3.1 → win) | LOD 8 wins under FA | optional: LOD 8 for FA users (PLAYER-TUNING note) |
|
||
| B8 | LambDynamicLights fancy / fastest | 4.41/181.8, 4.35/187.7 vs 4.32/184.6 | tie | keep C1 `fast` |
|
||
| B9 | Flywheel light flat / smooth (S2) | 3.45/246.6 (tie), 3.46/238.9 (−8.6, loss) vs 3.44/247.5 | tie/loss | keep C1 TRI_LINEAR |
|
||
| B10 | Sound Physics rays up / strict / no caps (S2s) | 3.69/216.2 (loss), 3.58/233.7 (tie), 3.64/216.9 (loss) vs 3.52/237.6 | loss/tie | keep C1 |
|
||
| B11 | Sodium 3 builder threads, blend 0/2, mipmaps 4 | all losses except blend 2 (S5 tie); e.g. mip4 S1 8.82/97.6 vs 8.12/104.2 | loss/tie | keep C1 |
|
||
| B12a | remove BadOptimizations / Particle Core / Sodium Extra | each loses ≥1 scene (e.g. S3 3.93/205.3, 4.63/188.4, 4.61/187.5 vs 3.87/212.5) | keep all | no change |
|
||
| B4h, B7 shader rows, B12b | heap ladder, shader profiles, trial mods | — | **pending run 2** | — |
|
||
| B13, B14 | spark profile; spawn-tuning counts | — | **pending (manual)** | — |
|
||
|
||
## Caveats
|
||
|
||
- **Single proxy machine.** Fleet machines (P0–P2) were unreachable; the fleet fps floors
|
||
(P0 1 % low ≥ 30, P1/P2 ≥ 60) don't map to a 7800 XT — every verdict here is relative.
|
||
B2's "C1 wins on ≥ 2 machines" gate is satisfied on one machine only.
|
||
- **This GPU hides render costs.** Airships (B5), Flywheel lighting and DH radius are cheap
|
||
here; the fleet's older GPUs may rank them differently. B5 decay is deferred to P0/P1.
|
||
- **GPU clocks unpinned** (amdgpu DPM `auto`, no root); every row carries `gpu_dpm_auto`.
|
||
- **JVM differed between block groups (harness bug).** B0–B3 ran Prism's default args at
|
||
8 GB; B4 set each GC at 6 GB; `bench.py` never restored `instance.cfg`, so **B5–B12a all
|
||
ran on J3 Shenandoah, 6 GB**. Every verdict compares sessions inside one block (same JVM),
|
||
so verdicts stand, but absolute numbers aren't comparable across those groups and the B0
|
||
noise floor was measured under the other JVM.
|
||
- **One crash**: B10 `spr-nocap`, first attempt — `OutOfMemoryError: Ran out of memory
|
||
reading level data` after an unusually slow (~1.5 min) resource load, on the leaked
|
||
Shenandoah/6 GB JVM. The automatic retry was clean.
|
||
- Single sessions per variant (except B0/B6, 3 each); 3 reps per scene per session.
|