chore(tools): add FTB Chunks -> Xaero's map reveal migration tooling

The swap does not carry players' explored terrain across. FTB Chunks stores no rendered map (its region zips hold chunks.dat plus five data PNGs, one of which packs height into an image channel), and Xaero's region format is closed and now version 6 with per-pixel NBT blobs, so the community converters no longer work. Rather than convert pixels, prune a copy of the world save down to exactly the chunks a player revealed and let Xaero's own generator map it. Validated against real data: extracting the 167-region / 98,170-chunk mask cross-checks clean against FTB's own pixel data, and Anvil pruning round-trips byte-identically (15.7 MB -> 471 KB on a live region file, slot-to-chunk mapping confirmed against 1024 chunk NBT headers). Deferred for now: needs the world save and one singleplayer pass per player. See analysis/FTB-XAERO-MAP-MIGRATION.md.
2026-09-30 22:04:56 -04:00
1 parent 1d02bf1cf6
commit 769af6fa2d
@@ -0,0 +1,139 @@
+# FTB Chunks -> Open Parties and Claims + Xaero's maps
+
+**Status:** implemented in the pack (working tree, uncommitted) - 2026-09-30
+**Decision:** replace FTB Chunks with **Open Parties and Claims (OPAC)** for claims/forceloading and
+**Xaero's World Map + Minimap** for mapping. FTB Teams, FTB Quests, FTB Library, FTB XMod Compat and
+FTB Backups 3 all stay.
+**Accepted cost:** every player re-claims their land and re-creates their waypoints.
+
+Confidence tags follow `FINDINGS.md`: **VERIFIED** / **DOCUMENTED** / **COMMUNITY** / **INFERENCE**.
+
+---
+
+## 1. Why this works at all
+
+The pack's earlier review (`MODLIST-REVIEW.md` §8) recommended keeping FTB Chunks on the grounds that
+"wholesale replacement breaks FTB Quests/Teams". That is only true of a *wholesale* FTB removal.
+**VERIFIED:** FTB Quests' build declares `ftb-library` + `ftb-teams` as required and `ftb-xmod-compat`
+as optional - **not** FTB Chunks. FTB Quests also ships **no** claim-chunk task type. So FTB Chunks is
+the one mod in the FTB suite that can leave without touching quests, provided Teams stays.
+
+**VERIFIED:** nothing else in the pack declares FTB Chunks as a required dependency. The only
+cross-mod hooks are inside FTB XMod Compat (KubeJS/GameStages progression, rank-based claim limits),
+which degrade to inert modules.
+
+## 2. What changed
+
+| Action | Path |
+| --- | --- |
+| Removed | `mods/ftb-chunks-forge.pw.toml` (was `ftb-chunks-neoforge-2101.1.22`, CF 314906) |
+| Removed | `config/ftbchunks-client.snbt` (the `minimap.enabled=true` modpack default) |
+| Edited | `.packwizignore` - dropped `config/*` + `!config/ftbchunks-client.snbt`, back to `config/` |
+| Added | `mods/open-parties-and-claims.pw.toml` - `neoforge-1.21.1-0.31.6` (`side = "both"`) |
+| Added | `mods/xaeros-world-map.pw.toml` - `neoforge-1.21.1-1.46.0` (`side = "both"`) |
+| Added | `mods/xaeros-minimap.pw.toml` - `neoforge-1.21.1-26.5.0` (`side = "both"`) |
+| Added | `mods/xaeros-minimap-world-map-waystones-compatibility-forge.pw.toml` - `2.1.0` |
+| Regenerated | `index.toml` (147 entries) and the `[index]` hash in `pack.toml` |
+
+Net: **146 mods** (138 Modrinth/other + 8 CurseForge), down from 9 CurseForge-exclusive mods. All four
+new mods are Modrinth-hosted, so they are checkable by `packwiz update`.
+
+The three Xaero's entries are byte-identical to the ones deleted in `b0542f9` (2026-09-28) - same
+versions, same sha512s - so this is a revert of the map half of that commit, not a new integration.
+
+## 3. Where the pack traded one thing for another
+
+| FTB Chunks capability | Replacement | Notes |
+| --- | --- | --- |
+| Chunk claims + protection | OPAC | Server config now lives in the world save, not the pack. |
+| Forceload + offline ticking | OPAC `claims.forceload.offlineForceload` | Farms-vs-TPS trade is unchanged; reliability is less proven (see §5). |
+| Minimap + fullscreen map | Xaero's Minimap + World Map | Now two client mods instead of one. |
+| Waypoints + teleport | Xaero's Minimap | **No migration path** - see §4. |
+| Party/team claims | OPAC `primaryPartySystem` = FTB Teams | Must be set, or OPAC runs its own parallel party system. |
+| Waystones on the map | `xaeros_waystones_compatibility` | Restored; it had been removed with the maps. |
+| FTB Ranks permission overrides | `xaero.pac_max_claims` / `xaero.pac_max_forceloads` | Moot in practice: FTB Ranks was never installed. |
+
+## 4. Migration runbook (server operator)
+
+**Before the update**
+1. Take a world backup. FTB Chunks' claim data stays in the save after the mod is removed, so this
+   backup is the rollback path.
+2. Have each player screenshot or export their FTB Chunks waypoints. They live client-side under
+   `local/ftbchunks/`; **there is no FTB Chunks -> Xaero's converter** (only JourneyMap -> Xaero tools
+   exist). Re-entry is manual.
+3. Capture who owned which claims/forceloads, so players have a checklist and an admin can pre-claim
+   using OPAC server-claim mode if desired.
+
+**Cutover (one update, low-activity window)**
+4. Do **not** run FTB Chunks and OPAC side by side - two claim systems means split protection logic.
+   Hard cutover.
+5. Start the server once with OPAC installed so it generates
+   `world/serverconfig/openpartiesandclaims-server.toml`, then **stop the server**.
+6. Edit the generated file (it carries `# Default:` comments for every key) and set at minimum:
+   - `maxPlayerClaims = 500` (matches the old FTB Chunks cap)
+   - `maxPlayerClaimForceloads = 25` (**decided 2026-09-30**; the stock default is lower, so leaving it
+     alone silently removes forceloading)
+   - `primaryPartySystem` -> FTB Teams
+   - a decision on `claims.forceload.offlineForceload` (farms while offline vs TPS)
+   - add `claims.forceload.offlineForceload` to `playerConfigurablePlayerConfigOptions` /
+     `opConfigurablePlayerConfigOptions` if players should be able to toggle their own
+7. Re-claim: each player re-claims their base and re-forceloads, via the OPAC UI (`'`) or the Xaero's
+   world map (right-click / drag-select).
+
+**After**
+8. Verify per-player forceloading actually keeps chunks ticking with the owner offline. This is the
+   known weak point (§5).
+9. Verify protection across the modded stack - especially Create contraptions, Sable ships, and
+   fake-player automation.
+10. Re-tier any rank/permission limits you were relying on.
+
+## 5. Risks and open items
+
+- **OPAC force-loading reliability is the one thing that can invalidate the swap.** Open upstream
+  issues report chunks marked forceloadable but never actually loaded (#855, #927, plus #237/#528).
+  **COMMUNITY.** Test before trusting; re-test on every OPAC bump. If it fails, this swap should be
+  rolled back - forceloading is the one feature the pack cannot work around.
+- **OPAC is self-described beta** and releases frequently (0.31.6, 2026-09-10). **DOCUMENTED.**
+- **OPAC and Xaero's are version-coupled** - OPAC's README tells users to keep the Xaero's mods on
+  their latest versions. Bump them together.
+- **Claim protection parity is unverified.** OPAC protects vanilla plus a supported-mod list (Create is
+  built in on Forge), and needs manual exception entries for anything else. Open reports exist for
+  creeper explosions (#731) and modded projectiles (#758) inside claims. **COMMUNITY.** The pack's
+  Apotheosis/Sophisticated/Ars Nouveau/Alex's Mobs surface needs a pass.
+- **Sable / Create Aeronautics.** A crash with Sable contraptions was reported (#873) and is **closed**,
+  but was not re-verified against 0.31.6. **COMMUNITY.** Ships are not covered by plain chunk claims
+  under either system; the pack has no ship-claim mod. Options if that matters:
+  `aeroclaims` ("Create Aeronautics: Claims", NeoForge 1.21.1, works with **OPAC or FTB**), or
+  `sable-protect`. Note that dropping FTB Chunks does close the door on `ftbchunksaerospace` and
+  `Create-Aeronautics-FTB-Chunks`.
+- **Licensing.** OPAC is **LGPL-3.0** - no pack conditions. Xaero's World Map and Minimap are
+  **All Rights Reserved**: because this pack is distributed from `git.ljones.me` and not from
+  CurseForge/Modrinth, the terms require an **easily accessible credit link to an official Xaero page**
+  in the pack. Add one. **VERIFIED** (from the mods' own "Modpacks and Other" sections).
+- **`side = "both"` for Xaero's** puts them on the server too. That buys reliable multi-world map
+  identity and server-side map profiles, and matches the pre-`b0542f9` configuration, but the server
+  then registers status effects that stop *vanilla* clients joining unless disabled in
+  `config/xaero/*`. All players get mods via packwiz, so this is not expected to matter.
+- **Vanilla-client / optional-client is actually improved** for claims: OPAC's client side is optional,
+  so a player without it is still protected and can use `/oclaims` chat commands.
+
+## 6. Rollback
+
+Re-add `mods/ftb-chunks-forge.pw.toml` (and the client config if the minimap default is wanted) and
+`packwiz refresh`. FTB Chunks' claim/forceload data is untouched in the world save, so existing claims
+resume. Then remove OPAC and the Xaero's mods. Keep the pre-swap world backup regardless.
+
+## 7. Docs that need follow-up
+
+- **Map/waypoint rescue from the old FTB data** - see `analysis/FTB-XAERO-MAP-MIGRATION.md` and the
+  `tools/ftbchunks-reveal.py` + `tools/prune-world-to-reveal.py` scripts. Mask extraction is validated;
+  the world-save pruning step is written and unit-tested but not yet run.
+
+- `FINDINGS.md` - §3.3, §7.1, §8 rows 2-3, appendix A and appendix B updated in this change; §1's
+  mod-count statistics were left at their earlier snapshot (a note was added).
+- `MODLIST-REVIEW.md` §7.4 and §8 still say Open Parties and Claims is a SKIP/duplication and "Keep
+  FTB". **Stale** - the swap is now the intended design.
+- `MODLIST-GROUPS.md` / `analysis/master-table.md` / `analysis/groups-table.md` carry the same stale
+  OPAC verdict.
+- Not done: re-adding `xaero-train-map` (was `Ov2r3aVY`, 1.0.0). It is unblocked again by this change and
+  the pack has Create trains; one-line addition if wanted.
@@ -0,0 +1,214 @@
+# FTB Chunks reveal -> Xaero's World Map: what transfers and how
+
+**Status:** mask extraction implemented and validated against real data; world-save pruning implemented
+and unit-tested; final step blocked on a copy of the server world save.
+**Goal:** give each player a native Xaero's map covering, as closely as possible, exactly the terrain
+they had revealed in FTB Chunks.
+Confidence tags as in `FINDINGS.md`.
+
+---
+
+## 1. Why we do not convert the pixels
+
+FTB Chunks does **not** store a rendered map image. Verified by decompiling the pack's exact build,
+`ftb-chunks-neoforge-2101.1.22.jar` (class strings in `MapManager`, `MapRegionData`, `MapDimension`,
+`MapChunk`):
+
+```
+<game>/local/ftbchunks/data/<serverUUID>/
+├── dimensions.txt, block_map.txt, biome_map.txt
+└── <dimension>/
+    └── <XXXXX-XXXXX>.zip          # region = 512x512 blocks; name = region coord + 60000, %05X-%05X
+        ├── chunks.dat             # the chunks this player has actually seen
+        ├── data.png               # NOT viewable: (height << 16) | (water|light|biome) per pixel
+        ├── foliage.png            # biome-tinted foliage RGB
+        ├── grass.png              # biome-tinted grass RGB
+        ├── water.png              # biome-tinted water RGB
+        └── blocks.png             # FTB's internal 24-bit block colour index
+```
+
+The colour a player actually saw is computed at runtime by combining `blocks.png`'s index with FTB's own
+`BlockColor` model via `block_map.txt`. Reproducing it means reimplementing FTB Chunks' renderer, and the
+result would look like FTB's map - i.e. worse than native Xaero's. So we take the **reveal footprint**
+from FTB Chunks and the **pixels** from Xaero's.
+
+## 2. Why we also do not write Xaero region files
+
+The current Xaero's region format is not the format the community tools understand. Comparing a real
+`region.xaero` from Xaero's World Map 1.46.0 (extracted from the client instance) with the format
+`JMtoXaero/RegionToXaero` writes:
+
+| | old format (RegionToXaero) | real 1.46.0 |
+| --- | --- | --- |
+| header | `FF` then int 4 | `FF 00 06 00 08` + varying byte (format 6) |
+| pixel payload | 4-byte parameters (+ optional state / colour / biome) | embeds NBT compounds |
+| size for region `0,0` | ~25 KB | **355 KB** |
+
+The 1.46.0 payload contains literal NBT - e.g. `0A 00 00 08 00 04 "Name" 00 10 "minecraft:gravel"` - so
+each pixel now carries a block descriptor. This matches `JMtoXaero`'s own admission that its region
+**merger** broke once Xaero changed the format. Writing this format is a live reverse-engineering project
+against a closed, obfuscated, moving target. Not worth it.
+
+**Therefore: drive Xaero's own generator, and control what it can see.**
+
+## 3. The mechanism
+
+Both mods use the same geometry - a region is 512x512 blocks = 32x32 chunks - so an FTB region maps 1:1
+onto one Anvil region file, and an FTB within-region chunk `(x,z)` maps 1:1 onto Anvil slot
+`x + z*32`. That makes the transplant exact rather than approximate:
+
+1. Extract each player's reveal set from their `chunks.dat` files.
+2. Prune a copy of the world save to exactly those chunks (Anvil header/sector surgery; chunk payloads
+   copied byte-for-byte, no NBT re-encoding).
+3. The player opens the pruned save in **singleplayer** with the pack; Xaero's maps it directly from the
+   save, so absent chunks simply never appear on the map.
+4. Copy the generated instance
+   (`xaero/world-map/World/null/<map>/`) into the server's folder
+   (`xaero/world-map/Multiplayer_advilplug.top/<DIM>/`).
+
+Result: **per-chunk-exact reveal extent**, rendered natively by Xaero's. Residual differences are only
+the colour palette/shading (Xaero's, not FTB's) - which is the whole point.
+
+## 4. What the real data says
+
+Measured on the supplied archive - **Stirwick**, `minecraft:overworld`:
+
+| Metric | Value |
+| --- | --- |
+| Region files | 167 |
+| Revealed chunks | **98,170** (of 171,008 in the bounding box) |
+| Region extent | x `-9..9`, z `-14..7` |
+| Block extent | x `-4608..5120`, z `-7168..4096` |
+| Fully revealed regions (`1024/1024`) | 10 |
+| Median region fill | 560 / 1024 (~55%) |
+| Sparse regions (`<64` chunks) | 10 |
+| `chunks.dat` vs `data.png` disagreements | **0** |
+
+Two consequences:
+
+- **`chunks.dat` alone is a trustworthy mask.** It was cross-checked against the `data.png` height
+  sentinel (`HeightUtils.UNKNOWN = Short.MIN_VALUE + 1 = -32767`) for all 167 regions. Zero
+  disagreements. Suspicion that claimed-but-never-visited chunks would inflate the set did not
+  materialise.
+- **Region-granularity pruning is not good enough.** The median region is only ~55% explored, and 10
+  regions are under 6%. Copying whole `.mca` files a player touched would reveal roughly twice the
+  terrain they had seen. Chunk-level pruning is worth the extra ~40 lines.
+
+`data.png` decoding detail, for anyone revisiting this: FTB writes `(height << 16) | wlb` into a
+`TYPE_INT_ARGB` image, so after PNG round-trip the 16-bit height is `(A << 8) | R` and the
+water/light/biome word is `(G << 8) | B`. Unrevealed pixels are exactly `A=0x80, R=0x01`.
+
+## 5. Tooling
+
+Both scripts are stdlib-only and live in `tools/` (which `.packwizignore` excludes, so they never ship).
+
+`tools/migrate-player.sh` - the driver, does steps 1-2 and prints the manual remainder:
+
+```
+tools/migrate-player.sh Stirwick <ftb-data-or-archive> <world-dir> \
+    --dimension minecraft:overworld --work /var/tmp/xaero-migration
+```
+
+`tools/ftbchunks-reveal.py` - **validated against Stirwick's real 167-region archive.**
+Emits a JSON mask of global chunk coordinates per dimension. Pass `--dimension` when the input is a
+bare directory/zip of region files, which cannot name its own dimension.
+
+```
+tools/ftbchunks-reveal.py <ftb-data-or-archive> --out mask.json --dimension minecraft:overworld --verify
+```
+
+`tools/filter-xaero-regions.py` - post-harvest cleanup. Compares the Xaero's map folder against the
+mask and removes regions the player never revealed (i.e. terrain Minecraft regenerated around the
+login position). File-name-only, so no Xaero format knowledge needed; reports before deleting.
+
+```
+tools/filter-xaero-regions.py <game>/xaero/world-map/Multiplayer_advilplug.top/null/ mask.json
+```
+
+`tools/prune-world-to-reveal.py` - **validated end-to-end against a real region file from the server**
+(`world/region/r.1.-2.mca`, 15.7 MB):
+
+| Check | Result |
+| --- | --- |
+| Anvil parse (1024 slots, all zlib, 2.5-21.8 KB payloads) | OK |
+| Slot -> chunk mapping (`slot = x + z*32`), confirmed by reading `xPos`/`zPos` NBT | **1024 / 1024 exact** |
+| Prune against that player's FTB mask (52 of 1024 chunks revealed) | 52 kept / 972 dropped |
+| Output sector-aligned | yes |
+| Kept slot set == mask | yes |
+| Kept payloads + timestamps byte-identical | yes |
+| Kept chunks re-parse with correct coordinates | 52 / 52 |
+| Size | 15,745,024 -> 471,040 bytes (3.0%) |
+
+Earlier synthetic tests also cover nested per-dimension `region`/`entities`/`poi` trees not being
+duplicated.
+
+```
+tools/prune-world-to-reveal.py <world-copy> mask.json <out-dir> --dimension minecraft:overworld
+```
+
+`--minimal` writes only `level.dat` plus the pruned `region/` tree - the smallest thing Xaero's can
+open and map, and the best fit when the full save is too large to copy around. `--dry-run` reports
+kept/dropped counts per region without writing anything. Runtime is ~0.15 s per 15 MB region.
+
+## 6. Running it where the save lives
+
+The tools are stdlib-only Python, so nothing has to be transferred: run them on the machine that holds
+the world. The live world is **not** on the VPS (the VPS is only a proxy in front of the real server),
+so this has to run where the save actually lives. The per-player footprint is only the pruned `region/`
+tree for the regions that player revealed, not the whole save.
+
+```bash
+tools/migrate-player.sh Stirwick <stirwick-ftbchunks-data> <world-dir> \
+    --dimension minecraft:overworld --work /var/tmp/xaero-migration
+```
+
+That extracts the mask (with the `chunks.dat` vs `data.png` cross-check) and prunes the world,
+dry-run first. Then, by hand:
+
+3. Copy the pruned world (`<work>/<player>/sp-world`) to a machine that can run the pack and open it in
+   **singleplayer** once. Let Xaero's map it, then quit.
+4. Move the generated instance, `xaero/world-map/World/null/<map>/`, into the server's map folder,
+   `xaero/world-map/Multiplayer_advilplug.top/<DIM>/`.
+5. Run `tools/filter-xaero-regions.py` on that folder to drop regions the player never revealed.
+6. Delete the scratch directory - usually a few hundred MB.
+
+### Two things that will bite
+
+- **The server does not need stopping, but freeze saves while it runs.** The pruner only reads the world
+  and writes to a separate output directory. However a running server rewrites region files on autosave
+  and rewrites `level.dat` in place, so a read can tear - and half a `level.dat` is an unopenable
+  singleplayer world. From the console: `save-off` -> dry run -> real run -> `save-all` -> `save-on`.
+  The pass takes a couple of minutes. Never forget `save-on`, and never point `--out` at the live world.
+  A region that does fail to read is reported, skipped, and left unwritten; re-run to retry it.
+
+- **Opening the world regenerates chunks near the login position.** Minecraft loads and generates the
+  chunks around wherever the player spawns, so terrain the player never revealed can reappear on the
+  map if it happens to sit near world spawn. It is bounded by render distance (well under one
+  512x512 region at typical settings), and it is easy to undo: step 4's cleanup compares the harvested
+  region file names against the mask and deletes any region the player never revealed. Keeping
+  `level.dat`'s spawn inside the revealed area, or not walking around in step 3, avoids the rest.
+- **Do not copy the world with hardlinks.** `cp -al` would make the singleplayer client's own writes to
+  `level.dat` land in the live server save. A real copy, or `--minimal` into a fresh directory, is safe.
+
+## 7. Open items
+
+- **Needs: the full world save, read in place on the server.** One real region file (`r.1.-2.mca`) has
+  now validated the reader, the slot mapping and the writer, so the remaining unknowns are operational
+  rather than format-related: total runtime over 167 regions (extrapolating ~0.15 s per 15 MB region,
+  a couple of minutes) and the size of the pruned output. A full 1024-chunk region on this server is
+  ~15 MB; the player revealed 57.4% of the bounding box overall, so expect the pruned per-player world
+  to land in the low hundreds of MB. It is scratch data and can be deleted once the Xaero's map has
+  been harvested.
+- **Per-player, per-dimension.** Only `minecraft:overworld` was supplied and only one player's data.
+  Modded dimensions will each need an FTB-data -> Xaero-dimension mapping. Xaero names its dimension
+  folders `null`/`DIM-1`/`DIM1` for vanilla and uses the `minecraft$overworld` form (`$`, not `:`) in
+  `dimension_config.txt`; a `dimensions.txt` with more than one entry will confirm the rest.
+- **Step 3 costs one client launch per player.** It cannot be done server-side; Xaero's maps from a save
+  on the client. ~10 launches for this pack.
+- **Waypoints are a separate, easier win.** FTB stores them in `waypoints.json` next to the region data
+  (empty, 16 bytes, for the sampled player). Xaero waypoints are plain files under `xaero/minimap/`,
+  so a converter is straightforward - worth doing at the same time if anyone has waypoints to keep.
+- **Fallbacks if the save never materialises:** region-granularity pruning (coarse, ~2x over-reveal), or
+  shipping the full "Convert All Regions" map to everyone (over-reveals everything, zero code).
+- Nothing here is time-sensitive: removing the mod from the pack does not touch `local/ftbchunks/`, so
+  every client's reveal data is still on disk and this remains doable later.