219 lines
14 KiB
Markdown
219 lines
14 KiB
Markdown
# Bench results — proxy machine, run 1 (B0–B12a)
|
||
|
||
**Proxy data only.** One machine (Ryzen 9 9900X / RX 7800 XT, Manjaro), not the fleet (P0 Deck,
|
||
P1 7700K+1060, P2 6700K+1080 Ti). Run 1: 2026-10-16 22:25 → ~2026-10-17 05:00 EDT, 56 sessions
|
||
(57 launches incl. one auto-retry), 306/309 capture windows valid. Source:
|
||
`~/onion-bench/results/session.jsonl` (`ts >= 2026-10-16T22:24`), verdicts from
|
||
`harness/analyze.py --baseline C1 --noise B0 --since 2026-10-16T22:24`.
|
||
Run 2 (B4h, B7 shader rows, B12b, B14) and Run 3 (new mod set, Flerovium confirmation,
|
||
B14 ×4) are below; B13 is not done.
|
||
|
||
## Method
|
||
|
||
- **Harness**: `harness/bench.py` + `matrix.py`. Every session is one cold launch at
|
||
1920×1080 in headless gamescope, world restored from `bench-world-v1*.zip` (DH LODs
|
||
pre-generated, `generatorPlan=DISABLED`), scenes driven by an in-world datapack, spectator
|
||
camera, FTB Backups off, dedicated server stopped.
|
||
- **Captures**: per scene 15 s warm-up + 30 s capture (S5 tour 120 s, no warm-up) × 3 reps,
|
||
45 s settle after join. Frametimes from MangoHud; metrics = p50/p99 frametime, 1 % low
|
||
(frametime-based), avg fps.
|
||
- **Win rule (BENCH-MATRIX §4)**: win = median p99 improves by more than the B0 p99 spread
|
||
AND 1 % low doesn't drop by more than the B0 1 %-low spread; loss = worse beyond either;
|
||
else tie (ties keep the current/upstream value).
|
||
- **Noise floor (B0, C0 × 3 sessions on S1)**: p99 spread **0.40 ms**, 1 %-low spread
|
||
**6.6 fps** (p99 9.64 ms, 1 % low 88.3 fps).
|
||
- **Baselines**: each variant is compared with the C1 session(s) of its own block
|
||
(`C1@B3`, `C1@B11.baseline`, …), i.e. same JVM and same time window.
|
||
|
||
## Highlights
|
||
|
||
- **The shipped tuning (C1) beats upstream defaults (C0) on every scene.** C0 (B1) vs C1
|
||
(B2), same JVM (Prism default, 8 GB):
|
||
|
||
| Scene | p99 C0 → C1 (ms) | 1 % low C0 → C1 (fps) | avg fps C0 → C1 |
|
||
|---|---|---|---|
|
||
| S1 vista | 9.33 → 8.23 | 93.0 → 104.9 | 218 → 272 |
|
||
| S2 Create floor | 3.79 → 3.22 | 218.7 → 259.8 | 694 → 975 |
|
||
| S3 mob pen | 7.43 → 4.46 | 110.1 → 186.0 | 255 → 395 |
|
||
| S4 airships | 2.93 → 2.44 | 298.0 → 341.3 | 789 → 1073 |
|
||
| S5 tour | 3.47 → 2.95 | 194.2 → 223.8 | 788 → 1017 |
|
||
|
||
Startup ("Game took"): C0 31.38 s → C1 25.36 s. Atlas: C0 4096×4096×4 → C1 4096×4096×2.
|
||
- **Distant Horizons is the biggest single render cost.** Turning DH off (B3.dh-off) gives
|
||
S1 p99 7.99 → 5.38 ms, 1 % low 105.5 → 144.0; S5 3.09 → 2.34 ms, 211.2 → 282.4.
|
||
DH's LOW quality preset at C1's radius 128 (`dh-qlow`) is a **win** on S1 (−0.96 ms,
|
||
+9.5 fps). SSAO costs 1.25 ms p99 on S1.
|
||
- **Fresh Animations is expensive.** S3 p99 4.32 → 8.65 ms (FA) / 10.20 ms (FA +
|
||
Extensions); 1 % low 184.6 → 104.9 / 89.0. It is opt-in (not enabled by default) — keep
|
||
it that way and document the cost.
|
||
- **The added performance mods earn their place.** Removing BadOptimizations, Particle Core
|
||
or Sodium Extra each loses (Particle Core / Sodium Extra: S3 1 % low −24 / −25 fps).
|
||
- **ModernFix `dynamic_resources` (B6)**: startup 7.6 s faster (26.32 vs 33.92 s median) and
|
||
better frametimes with it on. Ship is gated only on the manual missing-texture check
|
||
(see B6 row).
|
||
|
||
## Per-block verdicts
|
||
|
||
Numbers are p99 ms / 1 % low fps; Δ is vs the block's C1 baseline (+ = better p99).
|
||
|
||
| Block | Change tested | Result | Verdict | Ship action |
|
||
|---|---|---|---|---|
|
||
| B0 | noise floor, C0 × 3 on S1 | p99 9.64 (spread 0.40), 1 % low 88.3 (spread 6.6) | — | thresholds for all blocks |
|
||
| B1 | C0 baseline, S1–S5 | see highlights table; startup 31.38 s | reference | — |
|
||
| B2 | C1 vs C0 | C1 wins S1–S5 (table above) | **win** | C1 stays shipped; no bisect needed |
|
||
| B2 | C1-deck vs C1 | S1 7.08/115.0 vs 8.23/104.9 (win); S2/S4/S5 tie; S3 4.84/167.5 vs 4.46/186.0 (loss) | mixed | keep Deck tier as is (S3 loss: check `entityShadows`/particles on fleet P0) |
|
||
| B2 | C1-high vs C1 | S1 tie; S2–S5 losses (e.g. S4 2.66/311.6 vs 2.44/341.3) | expected cost | keep High as opt-in quality tier |
|
||
| B3 | DH radius 64 | S1 7.46/111.1 vs 7.99/105.5 (win); S5 tie | win S1 | already the Deck value; no change to C1 |
|
||
| B3 | DH radius 192 / 256 | S1 tie (7.69/105.5, 8.07/101.5); S5 tie | tie | **decision**: on this GPU 256 (upstream) ties 128; fleet floors decide — keep 128 until fleet data |
|
||
| B3 | DH LOW quality preset @ r128 (`dh-qlow`) | S1 7.03/115.0 vs 7.99/105.5 (+0.96, +9.5) | **win** (S1 only) | candidate for DEFAULT after an S5 run + visual check |
|
||
| B3 | DH SSAO on | S1 9.24/95.3 (−1.25, −10.2) | loss | confirms C1 SSAO off |
|
||
| B3 | DH threads 4 | S1 tie; S5 3.32/197.4 vs 3.09/211.2 (loss) | loss | threads stay 2 |
|
||
| B3 | DH off (reference) | S1 5.38/144.0; S5 2.34/282.4 | — | cost reference only |
|
||
| B4 | GC J0/J1/J2/J3 @ 6 GB, S2+S5 | S5: J0 3.07/226.8, J1 3.11/228.1 (tie), J2 3.40/216.6 (loss), J3 3.75/173.4 (loss); S2 all tie | **J0** | keep J0; heap ladder = B4h (run 2) |
|
||
| B5 | Sable ships 0/1/4 (proxy) | S4 3.00/280.6, 3.12/261.6, 2.97/273.2 | — | ships nearly free on this GPU |
|
||
| B5 | SSRD 32/64/128, sable-noshade (4 ships) | 3.02/265.0, 3.03/264.7, 3.11/261.9, 3.04/263.0 vs pooled C1 (−8 to −11 fps 1 % low) | loss/noise | keep C1 (SSRD 16, shading on); decay run deferred (fleet P0/P1) |
|
||
| B6 | ModernFix dynamic_resources off vs on (C1) | S1 9.11/83.8 vs 8.48/98.6; S2 3.29/236.1 vs 3.18/264.0; startup 33.92 vs 26.32 s | dynres on **wins** | ship `modernfix-mixins.properties` **after** the manual missing-texture check |
|
||
| B8 | Fresh Animations (+Ext) on S3 | 8.65/104.9 (FA), 10.20/89.0 (FA+Ext) vs 4.32/184.6 | large cost | keep opt-in; document in PLAYER-TUNING |
|
||
| B8 | EMF animation LOD 20 / 8 (with FA) | vs C1-fa (LOD 12) 8.65/104.9: LOD 20 8.91/101.8 (tie), LOD 8 8.16/108.0 (+0.49, +3.1 → win) | LOD 8 wins under FA | optional: LOD 8 for FA users (PLAYER-TUNING note) |
|
||
| B8 | LambDynamicLights fancy / fastest | 4.41/181.8, 4.35/187.7 vs 4.32/184.6 | tie | keep C1 `fast` |
|
||
| B9 | Flywheel light flat / smooth (S2) | 3.45/246.6 (tie), 3.46/238.9 (−8.6, loss) vs 3.44/247.5 | tie/loss | keep C1 TRI_LINEAR |
|
||
| B10 | Sound Physics rays up / strict / no caps (S2s) | 3.69/216.2 (loss), 3.58/233.7 (tie), 3.64/216.9 (loss) vs 3.52/237.6 | loss/tie | keep C1 |
|
||
| B11 | Sodium 3 builder threads, blend 0/2, mipmaps 4 | all losses except blend 2 (S5 tie); e.g. mip4 S1 8.82/97.6 vs 8.12/104.2 | loss/tie | keep C1 |
|
||
| B12a | remove BadOptimizations / Particle Core / Sodium Extra | each loses ≥1 scene (e.g. S3 3.93/205.3, 4.63/188.4, 4.61/187.5 vs 3.87/212.5) | keep all | no change |
|
||
| B4h, B7, B12b, B14 | see **Run 2** below | | | |
|
||
| B13 | spark profile | — | **pending (manual)** | — |
|
||
|
||
## Run 2 (2026-10-17/18): B4h, B7, B12b, B14
|
||
|
||
Same machine and method; **default JVM** (run 1's B5–B12a ran on a leaked J3/6 GB JVM —
|
||
harness bug fixed in `60e10b3`; within-block verdicts unaffected). Verdicts use the B0
|
||
thresholds (p99 0.40 ms, 1 % low 6.6 fps); one session per row, so treat single-scene
|
||
margins near the threshold as soft. Numbers: p99 ms / 1 % low fps.
|
||
|
||
### B4h — heap ladder (J0, S2 + S5)
|
||
|
||
| heap | S2 | S5 |
|
||
|---|---|---|
|
||
| 4 GB | 3.58 / 229.1 | 3.32 / 188.7 |
|
||
| 5 GB | 3.58 / 238.6 | 3.13 / 209.1 |
|
||
| 6 GB | 3.63 / 225.7 | 3.14 / 211.8 |
|
||
| 8 GB | 3.58 / 234.5 | 3.18 / 209.2 |
|
||
|
||
**Smallest heap within the winner's spread: 5 GB** (S5 1 % low 209.1 vs best 211.8).
|
||
4 GB loses on S5 (−23 fps). Heap is a launcher setting, not a shipped config →
|
||
PLAYER-TUNING recommendation (≥ 5 GB), not a `build.py` change.
|
||
|
||
### B7 — shaders (data for owner; no auto-decision)
|
||
|
||
| profile | S1 vegetation | S2 Create floor | S4 airships |
|
||
|---|---|---|---|
|
||
| none (run-2 re-run) | 8.13 / 103.6 (274 fps avg) | 3.28 / 260.4 (974) | 2.50 / 332.5 (1072) |
|
||
| MakeUp shadowless-medium | 8.69 / 97.3 (263) | 12.88 / 67.9 (120) | 2.58 / 314.0 (1050) |
|
||
| MakeUp medium | 10.04 / 88.4 (213) | 12.32 / 71.9 (121) | 2.94 / 280.4 (779) |
|
||
| Complementary LOW | 10.85 / 83.2 (182) | 13.23 / 64.0 (120) | 4.16 / 205.2 (444) |
|
||
| Complementary MEDIUM | 10.92 / 78.5 (175) | 13.27 / 67.4 (118) | 4.33 / 192.8 (401) |
|
||
|
||
All shader rows loaded the pack (no `shader_not_loaded`). The dominant cost is **S2**:
|
||
any shaderpack makes Flywheel drop its GPU backends, so the Create floor goes from
|
||
974 to ~120 fps average (p99 3.3 → ~13 ms) regardless of profile. Elsewhere MakeUp
|
||
shadowless is nearly free; Complementary costs ~2.7 ms p99 on S1 and ~2× frame time on S4.
|
||
→ Supports **opt-in shaders**, MakeUp first, and **Iris Flywheel Compat** (B12b).
|
||
|
||
### B12b — trial mods (each vs its own baseline)
|
||
|
||
| trial | result | verdict |
|
||
|---|---|---|
|
||
| **Iris Flywheel Compat** 2.4.0 (MakeUp default, vs shader-baseline) | S2 **3.98 / 218.3 vs 13.26 / 65.8** (avg 644 vs 117 fps); S1 9.92 / 84.8 vs 10.23 / 83.8 (tie); S4 3.10 / 266.1 vs 3.01 / 277.6 (1 % low −11.5, loss) | **win under shaders** (restores Flywheel instancing) — add after a visual check of Create contraptions/airships with a shaderpack (unofficial mod; breaks on Create/Iris updates) |
|
||
| **Flerovium** 1.1.2 (vs C1) | S3 **3.48 / 229.9 vs 5.71 / 147.4**; S1/S2/S5 tie | **win** on entities — but the B12b C1 S3 baseline (5.71) is worse than every run-1 C1 S3 (4.32–4.46), so the true gain is ~0.9 ms / +45 fps vs run 1. Add after one confirmation pair. **Superseded by Run 3** (did not hold up) |
|
||
| Gnetum 3.3.6 | S3 4.41 / 189.3 (vs anomalous 5.71; ≈ run-1 C1) ; S1/S2/S5 tie | tie in practice → no add |
|
||
| Better Block Entities 1.3.4 | S2 3.51 / 245.3 (loss), S3 loss; with FA 9.86 / 89.8 vs 9.52 / 87.4 (tie) | reject |
|
||
| Super Resolution 0.9.2-alpha.2 | S1 8.74 / 99.7 (loss), S5 1 % low −16.6 (loss) | reject on this proxy (P0-only lever; its first launch rewrites `fml.toml` and demands a restart) |
|
||
| Ixeris 4.6.8 | S1 p99 −0.49 (loss), S5 1 % low −12.2 (loss); **one crash** (NPE in `GameRules.getInt` during server tick, retry ok) | reject |
|
||
| Cull Leaves 4.1.1 (vs Better Leaves baseline) | S1 tie; S5 1 % low 208.8 vs 217.1 (loss) | reject |
|
||
|
||
New mod adds (Iris Flywheel Compat, Flerovium) ⇒ **re-run B0** before any later matrix work.
|
||
|
||
### B14 — spawn tuning (defaults vs shipped, 10 min per site, **1 run each**)
|
||
|
||
*Superseded by Run 3 (4 runs per config).*
|
||
|
||
| site | defaults total / tuned | shipped total / tuned |
|
||
|---|---|---|
|
||
| jungle (−8 107 −201) | 123 / 5 | 118 / 3 |
|
||
| river (904 63 88) | 208 / 29 | 208 / 25 |
|
||
| warm ocean (−200 63 56) | 155 / 22 | 130 / 0 |
|
||
|
||
Automated rule says **back off**, but it is driven by one Fowl Play herd: at the warm-ocean
|
||
site defaults spawned 14 geese + 8 ducks (on nearby land/water inside the 128-block
|
||
sphere), shipped none; the other "vanished" species (gazelle, dragonfly) had max count 1.
|
||
Sea bunny — the warm-ocean target — appeared under neither config, so that site doesn't
|
||
exercise it. **Inconclusive at n = 1**: needs ≥ 3 interleaved runs per config (≈ 3 h)
|
||
before any back-off; if it persists, the candidate is Fowl Play duck/goose weights.
|
||
All runs: 33 count rows, `biome_ok=1`, config check ok.
|
||
|
||
## Run 3 (2026-10-18): new mod set, Flerovium confirmation, B14 ×4
|
||
|
||
Same machine and method, default JVM, **new mod set** (shipped in `7841361`: + Flerovium
|
||
1.1.2, + Iris Flywheel Compat 2.4.0, ModernFix `dynamic_resources=true`). Numbers are
|
||
p99 ms / 1 % low fps, medians of session medians.
|
||
|
||
### B0v2 — noise floor for the new mod set (C0, S1, 3 sessions)
|
||
|
||
| p99 | p99 spread | 1 % low | 1 % low spread | avg fps |
|
||
|---|---|---|---|---|
|
||
| 9.74 | **0.41 ms** | 87.2 | **1.6 fps** | 219 |
|
||
|
||
These are the thresholds for any later matrix work on this mod set.
|
||
|
||
### Flerovium confirmation (C1, 2 sessions each, interleaved on/off)
|
||
|
||
| scene | Flerovium on | Flerovium off | Δ (on vs off) | verdict |
|
||
|---|---|---|---|---|
|
||
| S1 vista | 8.42 / 99.8 | 8.02 / 105.0 | p99 −0.40, 1 % low −5.2 | **loss** (1 % low drop > 1.6) |
|
||
| S3 mob pen | 5.55 / 151.9 | 5.94 / 141.1 | p99 +0.39 (just under 0.41) | **tie / inconclusive** — S3 session spread is huge (p99 1.42–2.34 ms, 1 % low 32–61 fps) |
|
||
|
||
**The run-2 win did not hold up** (it was measured against an anomalous S3 baseline).
|
||
**Recommendation: remove Flerovium** — awaiting owner OK; it is currently shipped in `7841361`.
|
||
|
||
### B14 — spawn tuning, 4 runs per config
|
||
|
||
Median final count at 10 min within 128 blocks (total / tuned species):
|
||
|
||
| site | defaults | shipped |
|
||
|---|---|---|
|
||
| jungle | 129.5 / 4.0 | 128.0 / 2.5 |
|
||
| river | 208.0 / 34.5 | 211.5 / 28.5 |
|
||
| warm ocean | 145.0 / 9.5 | 136.0 / 0.5 |
|
||
|
||
The automated rule says **back off** (snail vanished at the jungle site, gazelle at the warm
|
||
ocean), but both peaked at 1 per run — noise, not a disappearance. Sea bunny never appeared
|
||
under either config, so this site doesn't exercise it. The warm-ocean tuned drop is mostly
|
||
a Fowl Play goose/duck herd. **Verdict: the tuning is harmless but does not measurably
|
||
reduce total entity load.** Owner decides keep vs revert; no config change made.
|
||
|
||
### Shipping incident
|
||
|
||
`7841361`'s `index.toml` captured hashes of runtime re-saved spawn configs, so the server's
|
||
packwiz install failed (`Hash invalid`) at 14:06 and the server didn't start. Fixed by
|
||
re-hashing in `c5d25fa` (server up 14:18). Nothing was pushed.
|
||
|
||
## Caveats
|
||
|
||
- Run 2: Super Resolution's first launch flipped `earlyWindowControl` in the bench `fml.toml` (not journaled by the harness); later run-2 sessions started with it off. Loading-screen only — frametimes unaffected; startup times after 01:54 are not comparable. Reset 2026-10-18.
|
||
|
||
- **Single proxy machine.** Fleet machines (P0–P2) were unreachable; the fleet fps floors
|
||
(P0 1 % low ≥ 30, P1/P2 ≥ 60) don't map to a 7800 XT — every verdict here is relative.
|
||
B2's "C1 wins on ≥ 2 machines" gate is satisfied on one machine only.
|
||
- **This GPU hides render costs.** Airships (B5), Flywheel lighting and DH radius are cheap
|
||
here; the fleet's older GPUs may rank them differently. B5 decay is deferred to P0/P1.
|
||
- **GPU clocks unpinned** (amdgpu DPM `auto`, no root); every row carries `gpu_dpm_auto`.
|
||
- **JVM differed between block groups (harness bug).** B0–B3 ran Prism's default args at
|
||
8 GB; B4 set each GC at 6 GB; `bench.py` never restored `instance.cfg`, so **B5–B12a all
|
||
ran on J3 Shenandoah, 6 GB**. Every verdict compares sessions inside one block (same JVM),
|
||
so verdicts stand, but absolute numbers aren't comparable across those groups and the B0
|
||
noise floor was measured under the other JVM.
|
||
- **One crash**: B10 `spr-nocap`, first attempt — `OutOfMemoryError: Ran out of memory
|
||
reading level data` after an unusually slow (~1.5 min) resource load, on the leaked
|
||
Shenandoah/6 GB JVM. The automatic retry was clean.
|
||
- Single sessions per variant (except B0/B6, 3 each); 3 reps per scene per session.
|