diff --git a/analysis/FTB-CHUNKS-SWAP.md b/analysis/FTB-CHUNKS-SWAP.md new file mode 100644 index 0000000..df7b744 --- /dev/null +++ b/analysis/FTB-CHUNKS-SWAP.md @@ -0,0 +1,139 @@ +# FTB Chunks -> Open Parties and Claims + Xaero's maps + +**Status:** implemented in the pack (working tree, uncommitted) - 2026-09-30 +**Decision:** replace FTB Chunks with **Open Parties and Claims (OPAC)** for claims/forceloading and +**Xaero's World Map + Minimap** for mapping. FTB Teams, FTB Quests, FTB Library, FTB XMod Compat and +FTB Backups 3 all stay. +**Accepted cost:** every player re-claims their land and re-creates their waypoints. + +Confidence tags follow `FINDINGS.md`: **VERIFIED** / **DOCUMENTED** / **COMMUNITY** / **INFERENCE**. + +--- + +## 1. Why this works at all + +The pack's earlier review (`MODLIST-REVIEW.md` §8) recommended keeping FTB Chunks on the grounds that +"wholesale replacement breaks FTB Quests/Teams". That is only true of a *wholesale* FTB removal. +**VERIFIED:** FTB Quests' build declares `ftb-library` + `ftb-teams` as required and `ftb-xmod-compat` +as optional - **not** FTB Chunks. FTB Quests also ships **no** claim-chunk task type. So FTB Chunks is +the one mod in the FTB suite that can leave without touching quests, provided Teams stays. + +**VERIFIED:** nothing else in the pack declares FTB Chunks as a required dependency. The only +cross-mod hooks are inside FTB XMod Compat (KubeJS/GameStages progression, rank-based claim limits), +which degrade to inert modules. + +## 2. What changed + +| Action | Path | +| --- | --- | +| Removed | `mods/ftb-chunks-forge.pw.toml` (was `ftb-chunks-neoforge-2101.1.22`, CF 314906) | +| Removed | `config/ftbchunks-client.snbt` (the `minimap.enabled=true` modpack default) | +| Edited | `.packwizignore` - dropped `config/*` + `!config/ftbchunks-client.snbt`, back to `config/` | +| Added | `mods/open-parties-and-claims.pw.toml` - `neoforge-1.21.1-0.31.6` (`side = "both"`) | +| Added | `mods/xaeros-world-map.pw.toml` - `neoforge-1.21.1-1.46.0` (`side = "both"`) | +| Added | `mods/xaeros-minimap.pw.toml` - `neoforge-1.21.1-26.5.0` (`side = "both"`) | +| Added | `mods/xaeros-minimap-world-map-waystones-compatibility-forge.pw.toml` - `2.1.0` | +| Regenerated | `index.toml` (147 entries) and the `[index]` hash in `pack.toml` | + +Net: **146 mods** (138 Modrinth/other + 8 CurseForge), down from 9 CurseForge-exclusive mods. All four +new mods are Modrinth-hosted, so they are checkable by `packwiz update`. + +The three Xaero's entries are byte-identical to the ones deleted in `b0542f9` (2026-09-28) - same +versions, same sha512s - so this is a revert of the map half of that commit, not a new integration. + +## 3. Where the pack traded one thing for another + +| FTB Chunks capability | Replacement | Notes | +| --- | --- | --- | +| Chunk claims + protection | OPAC | Server config now lives in the world save, not the pack. | +| Forceload + offline ticking | OPAC `claims.forceload.offlineForceload` | Farms-vs-TPS trade is unchanged; reliability is less proven (see §5). | +| Minimap + fullscreen map | Xaero's Minimap + World Map | Now two client mods instead of one. | +| Waypoints + teleport | Xaero's Minimap | **No migration path** - see §4. | +| Party/team claims | OPAC `primaryPartySystem` = FTB Teams | Must be set, or OPAC runs its own parallel party system. | +| Waystones on the map | `xaeros_waystones_compatibility` | Restored; it had been removed with the maps. | +| FTB Ranks permission overrides | `xaero.pac_max_claims` / `xaero.pac_max_forceloads` | Moot in practice: FTB Ranks was never installed. | + +## 4. Migration runbook (server operator) + +**Before the update** +1. Take a world backup. FTB Chunks' claim data stays in the save after the mod is removed, so this + backup is the rollback path. +2. Have each player screenshot or export their FTB Chunks waypoints. They live client-side under + `local/ftbchunks/`; **there is no FTB Chunks -> Xaero's converter** (only JourneyMap -> Xaero tools + exist). Re-entry is manual. +3. Capture who owned which claims/forceloads, so players have a checklist and an admin can pre-claim + using OPAC server-claim mode if desired. + +**Cutover (one update, low-activity window)** +4. Do **not** run FTB Chunks and OPAC side by side - two claim systems means split protection logic. + Hard cutover. +5. Start the server once with OPAC installed so it generates + `world/serverconfig/openpartiesandclaims-server.toml`, then **stop the server**. +6. Edit the generated file (it carries `# Default:` comments for every key) and set at minimum: + - `maxPlayerClaims = 500` (matches the old FTB Chunks cap) + - `maxPlayerClaimForceloads = 25` (**decided 2026-09-30**; the stock default is lower, so leaving it + alone silently removes forceloading) + - `primaryPartySystem` -> FTB Teams + - a decision on `claims.forceload.offlineForceload` (farms while offline vs TPS) + - add `claims.forceload.offlineForceload` to `playerConfigurablePlayerConfigOptions` / + `opConfigurablePlayerConfigOptions` if players should be able to toggle their own +7. Re-claim: each player re-claims their base and re-forceloads, via the OPAC UI (`'`) or the Xaero's + world map (right-click / drag-select). + +**After** +8. Verify per-player forceloading actually keeps chunks ticking with the owner offline. This is the + known weak point (§5). +9. Verify protection across the modded stack - especially Create contraptions, Sable ships, and + fake-player automation. +10. Re-tier any rank/permission limits you were relying on. + +## 5. Risks and open items + +- **OPAC force-loading reliability is the one thing that can invalidate the swap.** Open upstream + issues report chunks marked forceloadable but never actually loaded (#855, #927, plus #237/#528). + **COMMUNITY.** Test before trusting; re-test on every OPAC bump. If it fails, this swap should be + rolled back - forceloading is the one feature the pack cannot work around. +- **OPAC is self-described beta** and releases frequently (0.31.6, 2026-09-10). **DOCUMENTED.** +- **OPAC and Xaero's are version-coupled** - OPAC's README tells users to keep the Xaero's mods on + their latest versions. Bump them together. +- **Claim protection parity is unverified.** OPAC protects vanilla plus a supported-mod list (Create is + built in on Forge), and needs manual exception entries for anything else. Open reports exist for + creeper explosions (#731) and modded projectiles (#758) inside claims. **COMMUNITY.** The pack's + Apotheosis/Sophisticated/Ars Nouveau/Alex's Mobs surface needs a pass. +- **Sable / Create Aeronautics.** A crash with Sable contraptions was reported (#873) and is **closed**, + but was not re-verified against 0.31.6. **COMMUNITY.** Ships are not covered by plain chunk claims + under either system; the pack has no ship-claim mod. Options if that matters: + `aeroclaims` ("Create Aeronautics: Claims", NeoForge 1.21.1, works with **OPAC or FTB**), or + `sable-protect`. Note that dropping FTB Chunks does close the door on `ftbchunksaerospace` and + `Create-Aeronautics-FTB-Chunks`. +- **Licensing.** OPAC is **LGPL-3.0** - no pack conditions. Xaero's World Map and Minimap are + **All Rights Reserved**: because this pack is distributed from `git.ljones.me` and not from + CurseForge/Modrinth, the terms require an **easily accessible credit link to an official Xaero page** + in the pack. Add one. **VERIFIED** (from the mods' own "Modpacks and Other" sections). +- **`side = "both"` for Xaero's** puts them on the server too. That buys reliable multi-world map + identity and server-side map profiles, and matches the pre-`b0542f9` configuration, but the server + then registers status effects that stop *vanilla* clients joining unless disabled in + `config/xaero/*`. All players get mods via packwiz, so this is not expected to matter. +- **Vanilla-client / optional-client is actually improved** for claims: OPAC's client side is optional, + so a player without it is still protected and can use `/oclaims` chat commands. + +## 6. Rollback + +Re-add `mods/ftb-chunks-forge.pw.toml` (and the client config if the minimap default is wanted) and +`packwiz refresh`. FTB Chunks' claim/forceload data is untouched in the world save, so existing claims +resume. Then remove OPAC and the Xaero's mods. Keep the pre-swap world backup regardless. + +## 7. Docs that need follow-up + +- **Map/waypoint rescue from the old FTB data** - see `analysis/FTB-XAERO-MAP-MIGRATION.md` and the + `tools/ftbchunks-reveal.py` + `tools/prune-world-to-reveal.py` scripts. Mask extraction is validated; + the world-save pruning step is written and unit-tested but not yet run. + +- `FINDINGS.md` - §3.3, §7.1, §8 rows 2-3, appendix A and appendix B updated in this change; §1's + mod-count statistics were left at their earlier snapshot (a note was added). +- `MODLIST-REVIEW.md` §7.4 and §8 still say Open Parties and Claims is a SKIP/duplication and "Keep + FTB". **Stale** - the swap is now the intended design. +- `MODLIST-GROUPS.md` / `analysis/master-table.md` / `analysis/groups-table.md` carry the same stale + OPAC verdict. +- Not done: re-adding `xaero-train-map` (was `Ov2r3aVY`, 1.0.0). It is unblocked again by this change and + the pack has Create trains; one-line addition if wanted. diff --git a/analysis/FTB-XAERO-MAP-MIGRATION.md b/analysis/FTB-XAERO-MAP-MIGRATION.md new file mode 100644 index 0000000..31e0dff --- /dev/null +++ b/analysis/FTB-XAERO-MAP-MIGRATION.md @@ -0,0 +1,214 @@ +# FTB Chunks reveal -> Xaero's World Map: what transfers and how + +**Status:** mask extraction implemented and validated against real data; world-save pruning implemented +and unit-tested; final step blocked on a copy of the server world save. +**Goal:** give each player a native Xaero's map covering, as closely as possible, exactly the terrain +they had revealed in FTB Chunks. +Confidence tags as in `FINDINGS.md`. + +--- + +## 1. Why we do not convert the pixels + +FTB Chunks does **not** store a rendered map image. Verified by decompiling the pack's exact build, +`ftb-chunks-neoforge-2101.1.22.jar` (class strings in `MapManager`, `MapRegionData`, `MapDimension`, +`MapChunk`): + +``` +/local/ftbchunks/data// +├── dimensions.txt, block_map.txt, biome_map.txt +└── / + └── .zip # region = 512x512 blocks; name = region coord + 60000, %05X-%05X + ├── chunks.dat # the chunks this player has actually seen + ├── data.png # NOT viewable: (height << 16) | (water|light|biome) per pixel + ├── foliage.png # biome-tinted foliage RGB + ├── grass.png # biome-tinted grass RGB + ├── water.png # biome-tinted water RGB + └── blocks.png # FTB's internal 24-bit block colour index +``` + +The colour a player actually saw is computed at runtime by combining `blocks.png`'s index with FTB's own +`BlockColor` model via `block_map.txt`. Reproducing it means reimplementing FTB Chunks' renderer, and the +result would look like FTB's map - i.e. worse than native Xaero's. So we take the **reveal footprint** +from FTB Chunks and the **pixels** from Xaero's. + +## 2. Why we also do not write Xaero region files + +The current Xaero's region format is not the format the community tools understand. Comparing a real +`region.xaero` from Xaero's World Map 1.46.0 (extracted from the client instance) with the format +`JMtoXaero/RegionToXaero` writes: + +| | old format (RegionToXaero) | real 1.46.0 | +| --- | --- | --- | +| header | `FF` then int 4 | `FF 00 06 00 08` + varying byte (format 6) | +| pixel payload | 4-byte parameters (+ optional state / colour / biome) | embeds NBT compounds | +| size for region `0,0` | ~25 KB | **355 KB** | + +The 1.46.0 payload contains literal NBT - e.g. `0A 00 00 08 00 04 "Name" 00 10 "minecraft:gravel"` - so +each pixel now carries a block descriptor. This matches `JMtoXaero`'s own admission that its region +**merger** broke once Xaero changed the format. Writing this format is a live reverse-engineering project +against a closed, obfuscated, moving target. Not worth it. + +**Therefore: drive Xaero's own generator, and control what it can see.** + +## 3. The mechanism + +Both mods use the same geometry - a region is 512x512 blocks = 32x32 chunks - so an FTB region maps 1:1 +onto one Anvil region file, and an FTB within-region chunk `(x,z)` maps 1:1 onto Anvil slot +`x + z*32`. That makes the transplant exact rather than approximate: + +1. Extract each player's reveal set from their `chunks.dat` files. +2. Prune a copy of the world save to exactly those chunks (Anvil header/sector surgery; chunk payloads + copied byte-for-byte, no NBT re-encoding). +3. The player opens the pruned save in **singleplayer** with the pack; Xaero's maps it directly from the + save, so absent chunks simply never appear on the map. +4. Copy the generated instance + (`xaero/world-map/World/null//`) into the server's folder + (`xaero/world-map/Multiplayer_advilplug.top//`). + +Result: **per-chunk-exact reveal extent**, rendered natively by Xaero's. Residual differences are only +the colour palette/shading (Xaero's, not FTB's) - which is the whole point. + +## 4. What the real data says + +Measured on the supplied archive - **Stirwick**, `minecraft:overworld`: + +| Metric | Value | +| --- | --- | +| Region files | 167 | +| Revealed chunks | **98,170** (of 171,008 in the bounding box) | +| Region extent | x `-9..9`, z `-14..7` | +| Block extent | x `-4608..5120`, z `-7168..4096` | +| Fully revealed regions (`1024/1024`) | 10 | +| Median region fill | 560 / 1024 (~55%) | +| Sparse regions (`<64` chunks) | 10 | +| `chunks.dat` vs `data.png` disagreements | **0** | + +Two consequences: + +- **`chunks.dat` alone is a trustworthy mask.** It was cross-checked against the `data.png` height + sentinel (`HeightUtils.UNKNOWN = Short.MIN_VALUE + 1 = -32767`) for all 167 regions. Zero + disagreements. Suspicion that claimed-but-never-visited chunks would inflate the set did not + materialise. +- **Region-granularity pruning is not good enough.** The median region is only ~55% explored, and 10 + regions are under 6%. Copying whole `.mca` files a player touched would reveal roughly twice the + terrain they had seen. Chunk-level pruning is worth the extra ~40 lines. + +`data.png` decoding detail, for anyone revisiting this: FTB writes `(height << 16) | wlb` into a +`TYPE_INT_ARGB` image, so after PNG round-trip the 16-bit height is `(A << 8) | R` and the +water/light/biome word is `(G << 8) | B`. Unrevealed pixels are exactly `A=0x80, R=0x01`. + +## 5. Tooling + +Both scripts are stdlib-only and live in `tools/` (which `.packwizignore` excludes, so they never ship). + +`tools/migrate-player.sh` - the driver, does steps 1-2 and prints the manual remainder: + +``` +tools/migrate-player.sh Stirwick \ + --dimension minecraft:overworld --work /var/tmp/xaero-migration +``` + +`tools/ftbchunks-reveal.py` - **validated against Stirwick's real 167-region archive.** +Emits a JSON mask of global chunk coordinates per dimension. Pass `--dimension` when the input is a +bare directory/zip of region files, which cannot name its own dimension. + +``` +tools/ftbchunks-reveal.py --out mask.json --dimension minecraft:overworld --verify +``` + +`tools/filter-xaero-regions.py` - post-harvest cleanup. Compares the Xaero's map folder against the +mask and removes regions the player never revealed (i.e. terrain Minecraft regenerated around the +login position). File-name-only, so no Xaero format knowledge needed; reports before deleting. + +``` +tools/filter-xaero-regions.py /xaero/world-map/Multiplayer_advilplug.top/null/ mask.json +``` + +`tools/prune-world-to-reveal.py` - **validated end-to-end against a real region file from the server** +(`world/region/r.1.-2.mca`, 15.7 MB): + +| Check | Result | +| --- | --- | +| Anvil parse (1024 slots, all zlib, 2.5-21.8 KB payloads) | OK | +| Slot -> chunk mapping (`slot = x + z*32`), confirmed by reading `xPos`/`zPos` NBT | **1024 / 1024 exact** | +| Prune against that player's FTB mask (52 of 1024 chunks revealed) | 52 kept / 972 dropped | +| Output sector-aligned | yes | +| Kept slot set == mask | yes | +| Kept payloads + timestamps byte-identical | yes | +| Kept chunks re-parse with correct coordinates | 52 / 52 | +| Size | 15,745,024 -> 471,040 bytes (3.0%) | + +Earlier synthetic tests also cover nested per-dimension `region`/`entities`/`poi` trees not being +duplicated. + +``` +tools/prune-world-to-reveal.py mask.json --dimension minecraft:overworld +``` + +`--minimal` writes only `level.dat` plus the pruned `region/` tree - the smallest thing Xaero's can +open and map, and the best fit when the full save is too large to copy around. `--dry-run` reports +kept/dropped counts per region without writing anything. Runtime is ~0.15 s per 15 MB region. + +## 6. Running it where the save lives + +The tools are stdlib-only Python, so nothing has to be transferred: run them on the machine that holds +the world. The live world is **not** on the VPS (the VPS is only a proxy in front of the real server), +so this has to run where the save actually lives. The per-player footprint is only the pruned `region/` +tree for the regions that player revealed, not the whole save. + +```bash +tools/migrate-player.sh Stirwick \ + --dimension minecraft:overworld --work /var/tmp/xaero-migration +``` + +That extracts the mask (with the `chunks.dat` vs `data.png` cross-check) and prunes the world, +dry-run first. Then, by hand: + +3. Copy the pruned world (`//sp-world`) to a machine that can run the pack and open it in + **singleplayer** once. Let Xaero's map it, then quit. +4. Move the generated instance, `xaero/world-map/World/null//`, into the server's map folder, + `xaero/world-map/Multiplayer_advilplug.top//`. +5. Run `tools/filter-xaero-regions.py` on that folder to drop regions the player never revealed. +6. Delete the scratch directory - usually a few hundred MB. + +### Two things that will bite + +- **The server does not need stopping, but freeze saves while it runs.** The pruner only reads the world + and writes to a separate output directory. However a running server rewrites region files on autosave + and rewrites `level.dat` in place, so a read can tear - and half a `level.dat` is an unopenable + singleplayer world. From the console: `save-off` -> dry run -> real run -> `save-all` -> `save-on`. + The pass takes a couple of minutes. Never forget `save-on`, and never point `--out` at the live world. + A region that does fail to read is reported, skipped, and left unwritten; re-run to retry it. + +- **Opening the world regenerates chunks near the login position.** Minecraft loads and generates the + chunks around wherever the player spawns, so terrain the player never revealed can reappear on the + map if it happens to sit near world spawn. It is bounded by render distance (well under one + 512x512 region at typical settings), and it is easy to undo: step 4's cleanup compares the harvested + region file names against the mask and deletes any region the player never revealed. Keeping + `level.dat`'s spawn inside the revealed area, or not walking around in step 3, avoids the rest. +- **Do not copy the world with hardlinks.** `cp -al` would make the singleplayer client's own writes to + `level.dat` land in the live server save. A real copy, or `--minimal` into a fresh directory, is safe. + +## 7. Open items + +- **Needs: the full world save, read in place on the server.** One real region file (`r.1.-2.mca`) has + now validated the reader, the slot mapping and the writer, so the remaining unknowns are operational + rather than format-related: total runtime over 167 regions (extrapolating ~0.15 s per 15 MB region, + a couple of minutes) and the size of the pruned output. A full 1024-chunk region on this server is + ~15 MB; the player revealed 57.4% of the bounding box overall, so expect the pruned per-player world + to land in the low hundreds of MB. It is scratch data and can be deleted once the Xaero's map has + been harvested. +- **Per-player, per-dimension.** Only `minecraft:overworld` was supplied and only one player's data. + Modded dimensions will each need an FTB-data -> Xaero-dimension mapping. Xaero names its dimension + folders `null`/`DIM-1`/`DIM1` for vanilla and uses the `minecraft$overworld` form (`$`, not `:`) in + `dimension_config.txt`; a `dimensions.txt` with more than one entry will confirm the rest. +- **Step 3 costs one client launch per player.** It cannot be done server-side; Xaero's maps from a save + on the client. ~10 launches for this pack. +- **Waypoints are a separate, easier win.** FTB stores them in `waypoints.json` next to the region data + (empty, 16 bytes, for the sampled player). Xaero waypoints are plain files under `xaero/minimap/`, + so a converter is straightforward - worth doing at the same time if anyone has waypoints to keep. +- **Fallbacks if the save never materialises:** region-granularity pruning (coarse, ~2x over-reveal), or + shipping the full "Convert All Regions" map to everyone (over-reveals everything, zero code). +- Nothing here is time-sensitive: removing the mod from the pack does not touch `local/ftbchunks/`, so + every client's reveal data is still on disk and this remains doable later. diff --git a/tools/filter-xaero-regions.py b/tools/filter-xaero-regions.py new file mode 100755 index 0000000..b18ace7 --- /dev/null +++ b/tools/filter-xaero-regions.py @@ -0,0 +1,88 @@ +#!/usr/bin/env python3 +"""Drop Xaero's map regions that a player never actually revealed. + +After a player opens their pruned singleplayer world, Xaero's maps it - but Minecraft +also *regenerates* the chunks around wherever that player logged in, so a little +terrain they had never seen can sneak onto the map. This compares the harvested map +against the reveal mask and removes the strays. + +It works on file names only (`_.zip` is one 512x512-block region, exactly the +same geometry as an FTB Chunks region file), so it needs no knowledge of Xaero's +region format. + +Usage: + filter-xaero-regions.py [--dimension ] [--delete] + +Without --delete it only reports. Run it on one dimension folder at a time, e.g. + ~/.minecraft/xaero/world-map/Multiplayer_advilplug.top/null/ +""" + +from __future__ import annotations + +import argparse +import json +import os +import re +import sys + +REGION_RE = re.compile(r"^(-?\d+)_(-?\d+)\.zip$") + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__.split("\n")[0]) + ap.add_argument("map_dir", help="Xaero's map instance directory holding _.zip") + ap.add_argument("mask", help="mask JSON from ftbchunks-reveal.py") + ap.add_argument("--dimension", default=None, + help="dimension to check (default: the mask's only dimension)") + ap.add_argument("--delete", action="store_true", help="actually remove the strays") + args = ap.parse_args() + + with open(args.mask) as fh: + mask = json.load(fh) + dims = mask["dimensions"] + dim = args.dimension + if dim is None: + if len(dims) != 1: + print(f"mask has {len(dims)} dimensions; pass --dimension", file=sys.stderr) + return 2 + dim = next(iter(dims)) + if dim not in dims: + print(f"no dimension '{dim}' in mask; have {list(dims)}", file=sys.stderr) + return 2 + + revealed = {tuple(int(v) for v in key.split(",")) for key in dims[dim]["regions"]} + + if not os.path.isdir(args.map_dir): + print(f"not a directory: {args.map_dir}", file=sys.stderr) + return 2 + + present, strays = [], [] + for name in sorted(os.listdir(args.map_dir)): + m = REGION_RE.match(name) + if not m: + continue + coords = (int(m.group(1)), int(m.group(2))) + (present if coords in revealed else strays).append((name, coords)) + + print(f"dimension {dim}") + print(f" map regions present : {len(present)} (of {len(revealed)} revealed)") + print(f" not in the mask : {len(strays)}") + for name, _ in strays: + print(f" {name}") + + missing = sorted(revealed - {c for _, c in present}) + if missing: + print(f" revealed but absent from the map: {len(missing)} " + f"(e.g. {missing[:4]}) - regenerate these, do not delete anything") + + if args.delete and strays: + for name, _ in strays: + os.remove(os.path.join(args.map_dir, name)) + print(f"\nremoved {len(strays)} stray region file(s)") + elif strays: + print("\nre-run with --delete to remove them") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/ftbchunks-reveal.py b/tools/ftbchunks-reveal.py new file mode 100755 index 0000000..e3f366b --- /dev/null +++ b/tools/ftbchunks-reveal.py @@ -0,0 +1,284 @@ +#!/usr/bin/env python3 +"""Extract a per-chunk reveal mask from FTB Chunks client map data. + +FTB Chunks stores its explored-terrain map client-side, one zip per 512x512-block +region: + + /local/ftbchunks/data///.zip + +Each zip holds `chunks.dat` (the chunks the player has actually seen) plus five +PNG data images. Region file names are the region coordinate + 60000, formatted +`%05X-%05X`, which is why overworld spawn shows up as `0EA60-0EA60.zip`. + +`chunks.dat` layout (verified against ftb-chunks-neoforge-2101.1.22 bytecode): + + u8 leading byte (0) + u8 version (3 on this build) + i16 big-endian chunk count + per chunk: + u8 chunk version (present when version >= 2) + u8 x within region (0..31) + u8 z within region (0..31) + i64 big-endian last-modified millis + +`chunks.dat` was checked against the `data.png` height sentinel (HeightUtils.UNKNOWN += -32767, packed as (height << 16) | (water|light|biome)) across all 167 regions of a +real 98,170-chunk dataset: zero disagreements. So chunks.dat alone is a trustworthy +reveal mask; --verify re-runs that cross-check anyway. + +Output is a JSON mask consumed by prune-world-to-reveal.py. + +Usage: + ftbchunks-reveal.py [--out mask.json] [--dimension ] [--verify] [--quiet] + + is either + * a client data directory, e.g. .../local/ftbchunks/data//, which is scanned + across all dimensions listed in its dimensions.txt, or + * a directory or zip containing region zips directly (a single dimension, as when + someone has archived just the overworld). Such an input cannot name its own + dimension, so pass --dimension to label it. + +Output coordinates are global chunk coordinates, so they are independent of which +region file a chunk happened to live in. +""" + +from __future__ import annotations + +import argparse +import io +import json +import os +import re +import struct +import sys +import zipfile +import zlib + +REGION_RE = re.compile(r"^([0-9A-Fa-f]{4,6})-([0-9A-Fa-f]{4,6})\.zip$") +REGION_OFFSET = 60000 + +# HeightUtils.UNKNOWN = Short.MIN_VALUE + 1, stored masked to 16 bits. +UNKNOWN_RAW = (-32767) & 0xFFFF + + +def parse_region_name(name: str) -> tuple[int, int] | None: + """`0EA60-0EA5F.zip` -> region coordinate (0, -1).""" + m = REGION_RE.match(os.path.basename(name)) + if not m: + return None + return int(m.group(1), 16) - REGION_OFFSET, int(m.group(2), 16) - REGION_OFFSET + + +def parse_chunks_dat(data: bytes) -> set[tuple[int, int]]: + """Return the within-region (x, z) chunk coordinates present in chunks.dat.""" + if len(data) < 4: + return set() + buf = io.BytesIO(data) + buf.read(1) + version = buf.read(1)[0] + count = struct.unpack(">h", buf.read(2))[0] + chunks: set[tuple[int, int]] = set() + for _ in range(count): + if version >= 2: + buf.read(1) + x = buf.read(1)[0] + z = buf.read(1)[0] + buf.read(8) + chunks.add((x, z)) + return chunks + + +# --- optional cross-check: decode data.png and find the UNKNOWN sentinel -------- + +def _png_decode(data: bytes) -> tuple[int, int, int, bytes]: + """Minimal PNG decoder (8-bit, non-interlaced) -> (w, h, channels, rgba bytes).""" + if data[:8] != b"\x89PNG\r\n\x1a\n": + raise ValueError("not a PNG") + pos, idat = 8, b"" + width = height = depth = colour = None + while pos < len(data): + (length,) = struct.unpack(">I", data[pos : pos + 4]) + kind = data[pos + 4 : pos + 8] + body = data[pos + 8 : pos + 8 + length] + pos += 12 + length + if kind == b"IHDR": + width, height, depth, colour, comp, filt, interlace = struct.unpack( + ">IIBBBBB", body + ) + if depth != 8 or interlace != 0: + raise ValueError(f"unsupported PNG depth/interlace {depth}/{interlace}") + elif kind == b"IDAT": + idat += body + elif kind == b"IEND": + break + channels = {0: 1, 2: 3, 3: 1, 4: 2, 6: 4}[colour] + stride = width * channels + raw = zlib.decompress(idat) + out = bytearray() + prev = bytearray(stride) + p = 0 + for _ in range(height): + ftype = raw[p] + p += 1 + line = bytearray(raw[p : p + stride]) + p += stride + if ftype == 1: + for i in range(channels, stride): + line[i] = (line[i] + line[i - channels]) & 0xFF + elif ftype == 2: + for i in range(stride): + line[i] = (line[i] + prev[i]) & 0xFF + elif ftype == 3: + for i in range(stride): + a = line[i - channels] if i >= channels else 0 + line[i] = (line[i] + ((a + prev[i]) >> 1)) & 0xFF + elif ftype == 4: + for i in range(stride): + a = line[i - channels] if i >= channels else 0 + b = prev[i] + c = prev[i - channels] if i >= channels else 0 + pp = a + b - c + pa, pb, pc = abs(pp - a), abs(pp - b), abs(pp - c) + pred = a if (pa <= pb and pa <= pc) else (b if pb <= pc else c) + line[i] = (line[i] + pred) & 0xFF + out += line + prev = line + return width, height, channels, bytes(out) + + +def chunks_seen_in_png(png_bytes: bytes) -> set[tuple[int, int]]: + """Chunks (0..31)^2 having at least one non-UNKNOWN pixel.""" + width, height, channels, px = _png_decode(png_bytes) + unknown_r = UNKNOWN_RAW & 0xFF + unknown_a = (UNKNOWN_RAW >> 8) & 0xFF + seen: set[tuple[int, int]] = set() + for cz in range(32): + for cx in range(32): + hit = False + for yy in range(16): + base = ((cz * 16 + yy) * width + cx * 16) * channels + for xx in range(16): + i = base + xx * channels + if not (px[i] == unknown_r and px[i + 3] == unknown_a): + hit = True + break + if hit: + break + if hit: + seen.add((cx, cz)) + return seen + + +# --- input discovery ----------------------------------------------------------- + +def iter_region_zips(path: str): + """Yield (dimension_name, region_filename, bytes) from a dir or an archive.""" + if zipfile.is_zipfile(path): + with zipfile.ZipFile(path) as outer: + for info in outer.infolist(): + if parse_region_name(info.filename) is None: + continue + yield "(unspecified)", os.path.basename(info.filename), outer.read(info) + return + if os.path.isdir(path) and any( + parse_region_name(n) for n in os.listdir(path) + ): + for name in sorted(os.listdir(path)): + if parse_region_name(name): + with open(os.path.join(path, name), "rb") as fh: + yield "(unspecified)", name, fh.read() + return + # treat as a client data directory keyed by dimension + dims_file = os.path.join(path, "dimensions.txt") + dims = [] + if os.path.exists(dims_file): + with open(dims_file) as fh: + dims = [ln.strip() for ln in fh if ln.strip()] + else: + dims = [ + d + for d in sorted(os.listdir(path)) + if os.path.isdir(os.path.join(path, d)) + ] + for dim in dims: + dim_dir = os.path.join(path, dim) + if not os.path.isdir(dim_dir): + continue + for name in sorted(os.listdir(dim_dir)): + if parse_region_name(name): + with open(os.path.join(dim_dir, name), "rb") as fh: + yield dim, name, fh.read() + + +def extract(path: str, verify: bool, quiet: bool, dimension_label: str | None = None) -> dict: + dims: dict[str, dict] = {} + total = 0 + mismatches = 0 + for dim, name, blob in iter_region_zips(path): + if dim == "(unspecified)" and dimension_label: + dim = dimension_label + coords = parse_region_name(name) + if coords is None: + continue + rx, rz = coords + with zipfile.ZipFile(io.BytesIO(blob)) as region: + listed = parse_chunks_dat(region.read("chunks.dat")) + if verify: + seen = chunks_seen_in_png(region.read("data.png")) + if seen != listed: + mismatches += 1 + print( + f" ! {dim}/{name}: chunks.dat={len(listed)} png={len(seen)}", + file=sys.stderr, + ) + entry = dims.setdefault(dim, {"regions": {}, "chunks": []}) + entry["regions"][f"{rx},{rz}"] = len(listed) + for cx, cz in sorted(listed): + entry["chunks"].append([rx * 32 + cx, rz * 32 + cz]) + total += len(listed) + if not quiet: + print(f" {dim:34} {name:>16} {len(listed):>4} chunks", file=sys.stderr) + + for entry in dims.values(): + entry["chunks"].sort() + entry["region_count"] = len(entry["regions"]) + entry["chunk_count"] = len(entry["chunks"]) + return { + "source": os.path.abspath(path), + "verified": verify, + "mismatches": mismatches, + "total_chunks": total, + "dimensions": dims, + } + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__.split("\n")[0]) + ap.add_argument("input", help="FTB Chunks data dir, or a dir/zip of region zips") + ap.add_argument("--out", default="ftb-reveal-mask.json") + ap.add_argument("--dimension", default=None, + help="dimension id used to label a bare directory/zip of region files, which " + "cannot say which dimension it is (e.g. minecraft:overworld)") + ap.add_argument("--verify", action="store_true", help="cross-check chunks.dat vs data.png") + ap.add_argument("--quiet", action="store_true") + args = ap.parse_args() + + mask = extract(args.input, args.verify, args.quiet, args.dimension) + with open(args.out, "w") as fh: + json.dump(mask, fh) + + if args.verify: + status = "OK" if mask["mismatches"] == 0 else f"{mask['mismatches']} MISMATCHES" + print(f"cross-check chunks.dat vs data.png: {status}", file=sys.stderr) + print( + f"{mask['total_chunks']:,} revealed chunks in " + f"{sum(d['region_count'] for d in mask['dimensions'].values())} regions " + f"across {len(mask['dimensions'])} dimension(s) -> {args.out}" + ) + for dim, d in mask["dimensions"].items(): + print(f" {dim}: {d['chunk_count']:,} chunks / {d['region_count']} regions") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/migrate-player.sh b/tools/migrate-player.sh new file mode 100755 index 0000000..631b17c --- /dev/null +++ b/tools/migrate-player.sh @@ -0,0 +1,131 @@ +#!/usr/bin/env bash +# End-to-end driver for the FTB Chunks -> Xaero's World Map reveal transplant. +# +# See analysis/FTB-XAERO-MAP-MIGRATION.md for the reasoning. This script does the +# parts that can be automated; the singleplayer pass in step 3 has to be done by hand +# on a machine that can run the pack. +# +# Usage: +# migrate-player.sh [--dimension ] [--work ] +# +# label used for output paths, e.g. Stirwick +# that player's local/ftbchunks/data//, or a bare zip/dir +# of region files, or an already-extracted mask .json +# the world save containing region/ +# +# Example: +# tools/migrate-player.sh Stirwick ~/Stirwick-ftbchunks ~/server/world \ +# --dimension minecraft:overworld --work /var/tmp/xaero-migration + +set -euo pipefail + +here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +if [ $# -lt 3 ]; then + sed -n '2,20p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' + exit 2 +fi + +player="$1"; ftb="$2"; world="$3"; shift 3 +dimension="minecraft:overworld" +work="$(pwd)/xaero-migration" + +while [ $# -gt 0 ]; do + case "$1" in + --dimension) dimension="$2"; shift 2 ;; + --work) work="$2"; shift 2 ;; + *) echo "unknown option: $1" >&2; exit 2 ;; + esac +done + +out="$work/$player" +mask="$out/mask.json" +sp="$out/sp-world" + +echo "player : $player" +echo "ftb data : $ftb" +echo "world : $world" +echo "dimension : $dimension" +echo "output : $out" +echo + +if [ ! -d "$world/region" ]; then + echo "error: $world/region not found" >&2 + exit 1 +fi +mkdir -p "$out" + +echo "== 1/2 extracting reveal mask ==" +if [ "${ftb##*.}" = "json" ]; then + cp -f "$ftb" "$mask" + echo " using prebuilt mask: $ftb" + python3 - "$mask" <<'PY' +import json, sys +m = json.load(open(sys.argv[1])) +dims = m["dimensions"] +print(f" {m['total_chunks']:,} chunks / " + f"{sum(d['region_count'] for d in dims.values())} regions / " + f"dimension(s) {list(dims)}") +PY +else + python3 "$here/ftbchunks-reveal.py" "$ftb" --out "$mask" --dimension "$dimension" --verify +fi + +echo +echo "== 2/2 pruning the world to those chunks (dry run first) ==" +python3 "$here/prune-world-to-reveal.py" "$world" "$mask" "$sp" \ + --dimension "$dimension" --minimal --dry-run +echo +python3 "$here/prune-world-to-reveal.py" "$world" "$mask" "$sp" \ + --dimension "$dimension" --minimal + +cat </saves/Stirwick-reveal/ + + Open it, open the map (M), let it fill, then quit. Xaero's maps what is in the + save, and this save holds only the revealed chunks. Do not walk around -- + Minecraft regenerates chunks near the login position and they will show up. + + Where the result lands: look under /xaero/world-map/ for a folder that + is not Multiplayer_*, e.g. World/null//. Sort by modification time if + unsure -- those files were just written. + + SANITY CHECK: you should end up with one zip per region in the mask + ($(python3 -c "import json;print(sum(d['region_count'] for d in json.load(open('$mask'))['dimensions'].values()))") for $player). A handful means it only mapped the spawn area and + you should use 'Convert All Regions' in the world map settings, or report back. + + 4. Clean the HARVEST before anything else -- this is the step that removes terrain + Minecraft regenerated around the login point. Run it on the harvested folder, + NOT on the player's own map folder: + + python3 $here/filter-xaero-regions.py \\ + "/xaero/world-map/World/null//" \\ + $mask --dimension $dimension + + Check the list, then re-run with --delete. + + 5. Merge the surviving _.zip files into the player's existing map folder, + WITHOUT overwriting anything already there (-n), so any terrain they have + explored since the swap is kept: + + cp -n /*.zip \\ + "/xaero/world-map/Multiplayer_advilplug.top/null//" + + The right instance is the mw\$... folder there. If the player has not joined the + server since the swap, have them join once first so Xaero's creates it. + + 6. Delete $out -- scratch, usually a few hundred MB. + + Per-dimension: repeat with --dimension and the matching map folder + (null = overworld, DIM-1 = nether, DIM1 = end). + Other players: each needs their own local/ftbchunks/data//; there is no + server-side copy of this data. +EOF diff --git a/tools/prune-world-to-reveal.py b/tools/prune-world-to-reveal.py new file mode 100755 index 0000000..20020fe --- /dev/null +++ b/tools/prune-world-to-reveal.py @@ -0,0 +1,210 @@ +#!/usr/bin/env python3 +"""Prune a Minecraft Anvil world save down to a set of revealed chunks. + +This is how we transplant FTB Chunks' reveal footprint onto Xaero's World Map +without reverse-engineering Xaero's region format: strip the world save down to +exactly the chunks a player had explored, then let Xaero's own generator map that +save in singleplayer. Xaero's renders it natively (correct palette, shading, +light, biome) and the absent chunks simply never appear on the map. + +Chunk payloads are copied through byte-for-byte -- no NBT re-encoding -- so the +output is safe for modded 1.21.1 chunk data. Only the 8 KiB header, the sector +offsets and the zero padding are rewritten. + +Usage: + prune-world-to-reveal.py + [--dimension minecraft:overworld] [--dry-run] + +Reads `/region/r...mca` and writes pruned copies to +`/region/`, along with whatever else the world needs. Nested per-dimension +`region`/`entities`/`poi` trees are never copied, so the Nether and End are not +duplicated at full size; `--minimal` goes further and copies only `level.dat`, which +is all Xaero's needs to map the overworld. + +If a full save copy is wanted (level.dat, data/, playerdata/ ...), copy the world +first and then run this in place with == . +""" + +from __future__ import annotations + +import argparse +import json +import os +import shutil +import struct +import sys + +SECTOR = 4096 +HEADER = 2 * SECTOR +COMPRESSION_NAMES = {1: "gzip", 2: "zlib", 3: "none", 4: "lz4", 127: "external"} + + +def read_slot(data: bytes, size: int, slot: int) -> tuple[bytes, int, int] | None: + """Return (payload, timestamp, compression) for an Anvil slot, or None if empty.""" + off = struct.unpack(">I", b"\x00" + data[slot * 4 : slot * 4 + 3])[0] + if off == 0: + return None + timestamp = struct.unpack(">I", data[SECTOR + slot * 4 : SECTOR + slot * 4 + 4])[0] + start = off * SECTOR + if start + 5 > len(data): + raise ValueError(f"slot {slot}: offset {off} past end of file") + (length,) = struct.unpack(">I", data[start : start + 4]) + compression = data[start + 4] + if length < 1 or start + 4 + length > len(data): + raise ValueError(f"slot {slot}: bad chunk length {length}") + return data[start + 5 : start + 4 + length], timestamp, compression + + +def prune_region(path: str, keep_slots: set[int], out_path: str, dry: bool) -> dict: + with open(path, "rb") as fh: + data = fh.read() + + kept: list[tuple[int, bytes, int, int]] = [] + dropped = 0 + external = 0 + for slot in range(1024): + entry = read_slot(data, SECTOR, slot) + if entry is None: + continue + payload, timestamp, compression = entry + if compression == 127: + external += 1 + if slot in keep_slots: + kept.append((slot, payload, timestamp, compression)) + else: + dropped += 1 + + stats = { + "file": os.path.basename(path), + "kept": len(kept), + "dropped": dropped, + "external_payloads": external, + } + if dry: + return stats + + out = bytearray(HEADER) + sector = 2 + for slot, payload, timestamp, compression in kept: + blob = struct.pack(">I", len(payload) + 1) + bytes([compression]) + payload + sectors = (len(blob) + SECTOR - 1) // SECTOR + if sector + sectors > 0xFFFFFF: + raise ValueError(f"{path}: region would exceed the Anvil 24-bit offset limit") + out[slot * 4 : slot * 4 + 3] = sector.to_bytes(3, "big") + out[slot * 4 + 3] = sectors + out[SECTOR + slot * 4 : SECTOR + slot * 4 + 4] = struct.pack(">I", timestamp) + out += blob + out += b"\x00" * (sectors * SECTOR - len(blob)) + sector += sectors + + os.makedirs(os.path.dirname(out_path) or ".", exist_ok=True) + with open(out_path, "wb") as fh: + fh.write(out) + return stats + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__.split("\n")[0]) + ap.add_argument("world", help="world save directory containing region/") + ap.add_argument("mask", help="mask JSON from ftbchunks-reveal.py") + ap.add_argument("out", help="output world directory (use the same path for in-place)") + ap.add_argument("--dimension", default="minecraft:overworld", + help="which dimension in the mask to apply (default overworld)") + ap.add_argument("--minimal", action="store_true", + help="output only level.dat + the pruned region/ tree (smallest usable world)") + ap.add_argument("--dry-run", action="store_true") + args = ap.parse_args() + + with open(args.mask) as fh: + mask = json.load(fh) + dims = mask["dimensions"] + if args.dimension not in dims: + print(f"mask has no dimension '{args.dimension}'; available: {list(dims)}", file=sys.stderr) + return 2 + + chunks = {(cx, cz) for cx, cz in dims[args.dimension]["chunks"]} + by_region: dict[tuple[int, int], set[int]] = {} + for cx, cz in chunks: + rx, rz = cx >> 5, cz >> 5 + by_region.setdefault((rx, rz), set()).add((cx & 31) + (cz & 31) * 32) + + in_region = os.path.join(args.world, "region") + if not os.path.isdir(in_region): + print(f"no region/ directory under {args.world}", file=sys.stderr) + return 2 + out_region = os.path.join(args.out, "region") + os.makedirs(out_region, exist_ok=True) + + # Copy the rest of the world so the pruned save is openable, unless in-place. + # Nested `region`/`entities`/`poi` trees (i.e. per-dimension ones like DIM-1/) are never + # copied - copying them wholesale would duplicate the Nether and End at full size. + if os.path.abspath(args.out) != os.path.abspath(args.world) and not args.dry_run: + ignore = shutil.ignore_patterns("region", "entities", "poi") + for name in sorted(os.listdir(args.world)): + if name in ("region", "entities", "poi"): + continue + if args.minimal and name not in ("level.dat", "level.dat_old"): + continue + src, dst = os.path.join(args.world, name), os.path.join(args.out, name) + if os.path.exists(dst): + continue + if os.path.isdir(src): + shutil.copytree(src, dst, ignore=ignore) + else: + shutil.copy2(src, dst) + + total_kept = total_dropped = 0 + seen_regions = set() + failures: list[tuple[str, str]] = [] + failed_regions: set[tuple[int, int]] = set() + for name in sorted(os.listdir(in_region)): + if not (name.startswith("r.") and name.endswith(".mca")): + continue + _, rx, rz, _ = name.split(".") + coords = (int(rx), int(rz)) + keep = by_region.get(coords) + if keep is None: + continue # region untouched by the reveal set + try: + stats = prune_region( + os.path.join(in_region, name), + keep, + os.path.join(out_region, name), + args.dry_run, + ) + except Exception as exc: + # A live server can rewrite a region file mid-read. A torn read is not a reason to + # abandon the whole pass - no output is written for a region that fails, so it can + # simply be retried. + failures.append((name, str(exc))) + failed_regions.add(coords) + print(f" ! {name}: {exc}", file=sys.stderr) + continue + seen_regions.add(coords) + total_kept += stats["kept"] + total_dropped += stats["dropped"] + if stats["external_payloads"]: + print(f" ! {name}: {stats['external_payloads']} external (.mcc) payload(s) " + f"referenced but not copied", file=sys.stderr) + print(f" {name}: kept {stats['kept']:>4} / dropped {stats['dropped']:>4}") + + # Any region the player revealed but that has no .mca file. + missing = sorted(set(by_region) - seen_regions - failed_regions) + if missing: + print(f" ! {len(missing)} revealed region(s) have no .mca in the save " + f"(e.g. {missing[:3]})", file=sys.stderr) + + verb = "would keep" if args.dry_run else "kept" + print(f"\n{verb} {total_kept:,} chunks, dropped {total_dropped:,}, " + f"across {len(seen_regions)} region file(s) -> {out_region}") + if failures: + print(f"\n{len(failures)} region(s) could not be read (not written):", file=sys.stderr) + for name, why in failures: + print(f" {name}: {why}", file=sys.stderr) + print("Re-run to retry them - usually a transient write by a running server.", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + sys.exit(main())