chore(tools): add FTB Chunks -> Xaero's map reveal migration tooling
The swap does not carry players' explored terrain across. FTB Chunks stores no rendered map (its region zips hold chunks.dat plus five data PNGs, one of which packs height into an image channel), and Xaero's region format is closed and now version 6 with per-pixel NBT blobs, so the community converters no longer work. Rather than convert pixels, prune a copy of the world save down to exactly the chunks a player revealed and let Xaero's own generator map it. Validated against real data: extracting the 167-region / 98,170-chunk mask cross-checks clean against FTB's own pixel data, and Anvil pruning round-trips byte-identically (15.7 MB -> 471 KB on a live region file, slot-to-chunk mapping confirmed against 1024 chunk NBT headers). Deferred for now: needs the world save and one singleplayer pass per player. See analysis/FTB-XAERO-MAP-MIGRATION.md.
This commit is contained in:
logan committed
1 parent
1d02bf1cf6
commit
769af6fa2d
6 files changed
+1066
No files matched your search
@@ -0,0 +1,139 @@
|
|||||||
|
# FTB Chunks -> Open Parties and Claims + Xaero's maps
|
||||||
|
|
||||||
|
**Status:** implemented in the pack (working tree, uncommitted) - 2026-09-30
|
||||||
|
**Decision:** replace FTB Chunks with **Open Parties and Claims (OPAC)** for claims/forceloading and
|
||||||
|
**Xaero's World Map + Minimap** for mapping. FTB Teams, FTB Quests, FTB Library, FTB XMod Compat and
|
||||||
|
FTB Backups 3 all stay.
|
||||||
|
**Accepted cost:** every player re-claims their land and re-creates their waypoints.
|
||||||
|
|
||||||
|
Confidence tags follow `FINDINGS.md`: **VERIFIED** / **DOCUMENTED** / **COMMUNITY** / **INFERENCE**.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Why this works at all
|
||||||
|
|
||||||
|
The pack's earlier review (`MODLIST-REVIEW.md` §8) recommended keeping FTB Chunks on the grounds that
|
||||||
|
"wholesale replacement breaks FTB Quests/Teams". That is only true of a *wholesale* FTB removal.
|
||||||
|
**VERIFIED:** FTB Quests' build declares `ftb-library` + `ftb-teams` as required and `ftb-xmod-compat`
|
||||||
|
as optional - **not** FTB Chunks. FTB Quests also ships **no** claim-chunk task type. So FTB Chunks is
|
||||||
|
the one mod in the FTB suite that can leave without touching quests, provided Teams stays.
|
||||||
|
|
||||||
|
**VERIFIED:** nothing else in the pack declares FTB Chunks as a required dependency. The only
|
||||||
|
cross-mod hooks are inside FTB XMod Compat (KubeJS/GameStages progression, rank-based claim limits),
|
||||||
|
which degrade to inert modules.
|
||||||
|
|
||||||
|
## 2. What changed
|
||||||
|
|
||||||
|
| Action | Path |
|
||||||
|
| --- | --- |
|
||||||
|
| Removed | `mods/ftb-chunks-forge.pw.toml` (was `ftb-chunks-neoforge-2101.1.22`, CF 314906) |
|
||||||
|
| Removed | `config/ftbchunks-client.snbt` (the `minimap.enabled=true` modpack default) |
|
||||||
|
| Edited | `.packwizignore` - dropped `config/*` + `!config/ftbchunks-client.snbt`, back to `config/` |
|
||||||
|
| Added | `mods/open-parties-and-claims.pw.toml` - `neoforge-1.21.1-0.31.6` (`side = "both"`) |
|
||||||
|
| Added | `mods/xaeros-world-map.pw.toml` - `neoforge-1.21.1-1.46.0` (`side = "both"`) |
|
||||||
|
| Added | `mods/xaeros-minimap.pw.toml` - `neoforge-1.21.1-26.5.0` (`side = "both"`) |
|
||||||
|
| Added | `mods/xaeros-minimap-world-map-waystones-compatibility-forge.pw.toml` - `2.1.0` |
|
||||||
|
| Regenerated | `index.toml` (147 entries) and the `[index]` hash in `pack.toml` |
|
||||||
|
|
||||||
|
Net: **146 mods** (138 Modrinth/other + 8 CurseForge), down from 9 CurseForge-exclusive mods. All four
|
||||||
|
new mods are Modrinth-hosted, so they are checkable by `packwiz update`.
|
||||||
|
|
||||||
|
The three Xaero's entries are byte-identical to the ones deleted in `b0542f9` (2026-09-28) - same
|
||||||
|
versions, same sha512s - so this is a revert of the map half of that commit, not a new integration.
|
||||||
|
|
||||||
|
## 3. Where the pack traded one thing for another
|
||||||
|
|
||||||
|
| FTB Chunks capability | Replacement | Notes |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Chunk claims + protection | OPAC | Server config now lives in the world save, not the pack. |
|
||||||
|
| Forceload + offline ticking | OPAC `claims.forceload.offlineForceload` | Farms-vs-TPS trade is unchanged; reliability is less proven (see §5). |
|
||||||
|
| Minimap + fullscreen map | Xaero's Minimap + World Map | Now two client mods instead of one. |
|
||||||
|
| Waypoints + teleport | Xaero's Minimap | **No migration path** - see §4. |
|
||||||
|
| Party/team claims | OPAC `primaryPartySystem` = FTB Teams | Must be set, or OPAC runs its own parallel party system. |
|
||||||
|
| Waystones on the map | `xaeros_waystones_compatibility` | Restored; it had been removed with the maps. |
|
||||||
|
| FTB Ranks permission overrides | `xaero.pac_max_claims` / `xaero.pac_max_forceloads` | Moot in practice: FTB Ranks was never installed. |
|
||||||
|
|
||||||
|
## 4. Migration runbook (server operator)
|
||||||
|
|
||||||
|
**Before the update**
|
||||||
|
1. Take a world backup. FTB Chunks' claim data stays in the save after the mod is removed, so this
|
||||||
|
backup is the rollback path.
|
||||||
|
2. Have each player screenshot or export their FTB Chunks waypoints. They live client-side under
|
||||||
|
`local/ftbchunks/`; **there is no FTB Chunks -> Xaero's converter** (only JourneyMap -> Xaero tools
|
||||||
|
exist). Re-entry is manual.
|
||||||
|
3. Capture who owned which claims/forceloads, so players have a checklist and an admin can pre-claim
|
||||||
|
using OPAC server-claim mode if desired.
|
||||||
|
|
||||||
|
**Cutover (one update, low-activity window)**
|
||||||
|
4. Do **not** run FTB Chunks and OPAC side by side - two claim systems means split protection logic.
|
||||||
|
Hard cutover.
|
||||||
|
5. Start the server once with OPAC installed so it generates
|
||||||
|
`world/serverconfig/openpartiesandclaims-server.toml`, then **stop the server**.
|
||||||
|
6. Edit the generated file (it carries `# Default:` comments for every key) and set at minimum:
|
||||||
|
- `maxPlayerClaims = 500` (matches the old FTB Chunks cap)
|
||||||
|
- `maxPlayerClaimForceloads = 25` (**decided 2026-09-30**; the stock default is lower, so leaving it
|
||||||
|
alone silently removes forceloading)
|
||||||
|
- `primaryPartySystem` -> FTB Teams
|
||||||
|
- a decision on `claims.forceload.offlineForceload` (farms while offline vs TPS)
|
||||||
|
- add `claims.forceload.offlineForceload` to `playerConfigurablePlayerConfigOptions` /
|
||||||
|
`opConfigurablePlayerConfigOptions` if players should be able to toggle their own
|
||||||
|
7. Re-claim: each player re-claims their base and re-forceloads, via the OPAC UI (`'`) or the Xaero's
|
||||||
|
world map (right-click / drag-select).
|
||||||
|
|
||||||
|
**After**
|
||||||
|
8. Verify per-player forceloading actually keeps chunks ticking with the owner offline. This is the
|
||||||
|
known weak point (§5).
|
||||||
|
9. Verify protection across the modded stack - especially Create contraptions, Sable ships, and
|
||||||
|
fake-player automation.
|
||||||
|
10. Re-tier any rank/permission limits you were relying on.
|
||||||
|
|
||||||
|
## 5. Risks and open items
|
||||||
|
|
||||||
|
- **OPAC force-loading reliability is the one thing that can invalidate the swap.** Open upstream
|
||||||
|
issues report chunks marked forceloadable but never actually loaded (#855, #927, plus #237/#528).
|
||||||
|
**COMMUNITY.** Test before trusting; re-test on every OPAC bump. If it fails, this swap should be
|
||||||
|
rolled back - forceloading is the one feature the pack cannot work around.
|
||||||
|
- **OPAC is self-described beta** and releases frequently (0.31.6, 2026-09-10). **DOCUMENTED.**
|
||||||
|
- **OPAC and Xaero's are version-coupled** - OPAC's README tells users to keep the Xaero's mods on
|
||||||
|
their latest versions. Bump them together.
|
||||||
|
- **Claim protection parity is unverified.** OPAC protects vanilla plus a supported-mod list (Create is
|
||||||
|
built in on Forge), and needs manual exception entries for anything else. Open reports exist for
|
||||||
|
creeper explosions (#731) and modded projectiles (#758) inside claims. **COMMUNITY.** The pack's
|
||||||
|
Apotheosis/Sophisticated/Ars Nouveau/Alex's Mobs surface needs a pass.
|
||||||
|
- **Sable / Create Aeronautics.** A crash with Sable contraptions was reported (#873) and is **closed**,
|
||||||
|
but was not re-verified against 0.31.6. **COMMUNITY.** Ships are not covered by plain chunk claims
|
||||||
|
under either system; the pack has no ship-claim mod. Options if that matters:
|
||||||
|
`aeroclaims` ("Create Aeronautics: Claims", NeoForge 1.21.1, works with **OPAC or FTB**), or
|
||||||
|
`sable-protect`. Note that dropping FTB Chunks does close the door on `ftbchunksaerospace` and
|
||||||
|
`Create-Aeronautics-FTB-Chunks`.
|
||||||
|
- **Licensing.** OPAC is **LGPL-3.0** - no pack conditions. Xaero's World Map and Minimap are
|
||||||
|
**All Rights Reserved**: because this pack is distributed from `git.ljones.me` and not from
|
||||||
|
CurseForge/Modrinth, the terms require an **easily accessible credit link to an official Xaero page**
|
||||||
|
in the pack. Add one. **VERIFIED** (from the mods' own "Modpacks and Other" sections).
|
||||||
|
- **`side = "both"` for Xaero's** puts them on the server too. That buys reliable multi-world map
|
||||||
|
identity and server-side map profiles, and matches the pre-`b0542f9` configuration, but the server
|
||||||
|
then registers status effects that stop *vanilla* clients joining unless disabled in
|
||||||
|
`config/xaero/*`. All players get mods via packwiz, so this is not expected to matter.
|
||||||
|
- **Vanilla-client / optional-client is actually improved** for claims: OPAC's client side is optional,
|
||||||
|
so a player without it is still protected and can use `/oclaims` chat commands.
|
||||||
|
|
||||||
|
## 6. Rollback
|
||||||
|
|
||||||
|
Re-add `mods/ftb-chunks-forge.pw.toml` (and the client config if the minimap default is wanted) and
|
||||||
|
`packwiz refresh`. FTB Chunks' claim/forceload data is untouched in the world save, so existing claims
|
||||||
|
resume. Then remove OPAC and the Xaero's mods. Keep the pre-swap world backup regardless.
|
||||||
|
|
||||||
|
## 7. Docs that need follow-up
|
||||||
|
|
||||||
|
- **Map/waypoint rescue from the old FTB data** - see `analysis/FTB-XAERO-MAP-MIGRATION.md` and the
|
||||||
|
`tools/ftbchunks-reveal.py` + `tools/prune-world-to-reveal.py` scripts. Mask extraction is validated;
|
||||||
|
the world-save pruning step is written and unit-tested but not yet run.
|
||||||
|
|
||||||
|
- `FINDINGS.md` - §3.3, §7.1, §8 rows 2-3, appendix A and appendix B updated in this change; §1's
|
||||||
|
mod-count statistics were left at their earlier snapshot (a note was added).
|
||||||
|
- `MODLIST-REVIEW.md` §7.4 and §8 still say Open Parties and Claims is a SKIP/duplication and "Keep
|
||||||
|
FTB". **Stale** - the swap is now the intended design.
|
||||||
|
- `MODLIST-GROUPS.md` / `analysis/master-table.md` / `analysis/groups-table.md` carry the same stale
|
||||||
|
OPAC verdict.
|
||||||
|
- Not done: re-adding `xaero-train-map` (was `Ov2r3aVY`, 1.0.0). It is unblocked again by this change and
|
||||||
|
the pack has Create trains; one-line addition if wanted.
|
||||||
@@ -0,0 +1,214 @@
|
|||||||
|
# FTB Chunks reveal -> Xaero's World Map: what transfers and how
|
||||||
|
|
||||||
|
**Status:** mask extraction implemented and validated against real data; world-save pruning implemented
|
||||||
|
and unit-tested; final step blocked on a copy of the server world save.
|
||||||
|
**Goal:** give each player a native Xaero's map covering, as closely as possible, exactly the terrain
|
||||||
|
they had revealed in FTB Chunks.
|
||||||
|
Confidence tags as in `FINDINGS.md`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Why we do not convert the pixels
|
||||||
|
|
||||||
|
FTB Chunks does **not** store a rendered map image. Verified by decompiling the pack's exact build,
|
||||||
|
`ftb-chunks-neoforge-2101.1.22.jar` (class strings in `MapManager`, `MapRegionData`, `MapDimension`,
|
||||||
|
`MapChunk`):
|
||||||
|
|
||||||
|
```
|
||||||
|
<game>/local/ftbchunks/data/<serverUUID>/
|
||||||
|
├── dimensions.txt, block_map.txt, biome_map.txt
|
||||||
|
└── <dimension>/
|
||||||
|
└── <XXXXX-XXXXX>.zip # region = 512x512 blocks; name = region coord + 60000, %05X-%05X
|
||||||
|
├── chunks.dat # the chunks this player has actually seen
|
||||||
|
├── data.png # NOT viewable: (height << 16) | (water|light|biome) per pixel
|
||||||
|
├── foliage.png # biome-tinted foliage RGB
|
||||||
|
├── grass.png # biome-tinted grass RGB
|
||||||
|
├── water.png # biome-tinted water RGB
|
||||||
|
└── blocks.png # FTB's internal 24-bit block colour index
|
||||||
|
```
|
||||||
|
|
||||||
|
The colour a player actually saw is computed at runtime by combining `blocks.png`'s index with FTB's own
|
||||||
|
`BlockColor` model via `block_map.txt`. Reproducing it means reimplementing FTB Chunks' renderer, and the
|
||||||
|
result would look like FTB's map - i.e. worse than native Xaero's. So we take the **reveal footprint**
|
||||||
|
from FTB Chunks and the **pixels** from Xaero's.
|
||||||
|
|
||||||
|
## 2. Why we also do not write Xaero region files
|
||||||
|
|
||||||
|
The current Xaero's region format is not the format the community tools understand. Comparing a real
|
||||||
|
`region.xaero` from Xaero's World Map 1.46.0 (extracted from the client instance) with the format
|
||||||
|
`JMtoXaero/RegionToXaero` writes:
|
||||||
|
|
||||||
|
| | old format (RegionToXaero) | real 1.46.0 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| header | `FF` then int 4 | `FF 00 06 00 08` + varying byte (format 6) |
|
||||||
|
| pixel payload | 4-byte parameters (+ optional state / colour / biome) | embeds NBT compounds |
|
||||||
|
| size for region `0,0` | ~25 KB | **355 KB** |
|
||||||
|
|
||||||
|
The 1.46.0 payload contains literal NBT - e.g. `0A 00 00 08 00 04 "Name" 00 10 "minecraft:gravel"` - so
|
||||||
|
each pixel now carries a block descriptor. This matches `JMtoXaero`'s own admission that its region
|
||||||
|
**merger** broke once Xaero changed the format. Writing this format is a live reverse-engineering project
|
||||||
|
against a closed, obfuscated, moving target. Not worth it.
|
||||||
|
|
||||||
|
**Therefore: drive Xaero's own generator, and control what it can see.**
|
||||||
|
|
||||||
|
## 3. The mechanism
|
||||||
|
|
||||||
|
Both mods use the same geometry - a region is 512x512 blocks = 32x32 chunks - so an FTB region maps 1:1
|
||||||
|
onto one Anvil region file, and an FTB within-region chunk `(x,z)` maps 1:1 onto Anvil slot
|
||||||
|
`x + z*32`. That makes the transplant exact rather than approximate:
|
||||||
|
|
||||||
|
1. Extract each player's reveal set from their `chunks.dat` files.
|
||||||
|
2. Prune a copy of the world save to exactly those chunks (Anvil header/sector surgery; chunk payloads
|
||||||
|
copied byte-for-byte, no NBT re-encoding).
|
||||||
|
3. The player opens the pruned save in **singleplayer** with the pack; Xaero's maps it directly from the
|
||||||
|
save, so absent chunks simply never appear on the map.
|
||||||
|
4. Copy the generated instance
|
||||||
|
(`xaero/world-map/World/null/<map>/`) into the server's folder
|
||||||
|
(`xaero/world-map/Multiplayer_advilplug.top/<DIM>/`).
|
||||||
|
|
||||||
|
Result: **per-chunk-exact reveal extent**, rendered natively by Xaero's. Residual differences are only
|
||||||
|
the colour palette/shading (Xaero's, not FTB's) - which is the whole point.
|
||||||
|
|
||||||
|
## 4. What the real data says
|
||||||
|
|
||||||
|
Measured on the supplied archive - **Stirwick**, `minecraft:overworld`:
|
||||||
|
|
||||||
|
| Metric | Value |
|
||||||
|
| --- | --- |
|
||||||
|
| Region files | 167 |
|
||||||
|
| Revealed chunks | **98,170** (of 171,008 in the bounding box) |
|
||||||
|
| Region extent | x `-9..9`, z `-14..7` |
|
||||||
|
| Block extent | x `-4608..5120`, z `-7168..4096` |
|
||||||
|
| Fully revealed regions (`1024/1024`) | 10 |
|
||||||
|
| Median region fill | 560 / 1024 (~55%) |
|
||||||
|
| Sparse regions (`<64` chunks) | 10 |
|
||||||
|
| `chunks.dat` vs `data.png` disagreements | **0** |
|
||||||
|
|
||||||
|
Two consequences:
|
||||||
|
|
||||||
|
- **`chunks.dat` alone is a trustworthy mask.** It was cross-checked against the `data.png` height
|
||||||
|
sentinel (`HeightUtils.UNKNOWN = Short.MIN_VALUE + 1 = -32767`) for all 167 regions. Zero
|
||||||
|
disagreements. Suspicion that claimed-but-never-visited chunks would inflate the set did not
|
||||||
|
materialise.
|
||||||
|
- **Region-granularity pruning is not good enough.** The median region is only ~55% explored, and 10
|
||||||
|
regions are under 6%. Copying whole `.mca` files a player touched would reveal roughly twice the
|
||||||
|
terrain they had seen. Chunk-level pruning is worth the extra ~40 lines.
|
||||||
|
|
||||||
|
`data.png` decoding detail, for anyone revisiting this: FTB writes `(height << 16) | wlb` into a
|
||||||
|
`TYPE_INT_ARGB` image, so after PNG round-trip the 16-bit height is `(A << 8) | R` and the
|
||||||
|
water/light/biome word is `(G << 8) | B`. Unrevealed pixels are exactly `A=0x80, R=0x01`.
|
||||||
|
|
||||||
|
## 5. Tooling
|
||||||
|
|
||||||
|
Both scripts are stdlib-only and live in `tools/` (which `.packwizignore` excludes, so they never ship).
|
||||||
|
|
||||||
|
`tools/migrate-player.sh` - the driver, does steps 1-2 and prints the manual remainder:
|
||||||
|
|
||||||
|
```
|
||||||
|
tools/migrate-player.sh Stirwick <ftb-data-or-archive> <world-dir> \
|
||||||
|
--dimension minecraft:overworld --work /var/tmp/xaero-migration
|
||||||
|
```
|
||||||
|
|
||||||
|
`tools/ftbchunks-reveal.py` - **validated against Stirwick's real 167-region archive.**
|
||||||
|
Emits a JSON mask of global chunk coordinates per dimension. Pass `--dimension` when the input is a
|
||||||
|
bare directory/zip of region files, which cannot name its own dimension.
|
||||||
|
|
||||||
|
```
|
||||||
|
tools/ftbchunks-reveal.py <ftb-data-or-archive> --out mask.json --dimension minecraft:overworld --verify
|
||||||
|
```
|
||||||
|
|
||||||
|
`tools/filter-xaero-regions.py` - post-harvest cleanup. Compares the Xaero's map folder against the
|
||||||
|
mask and removes regions the player never revealed (i.e. terrain Minecraft regenerated around the
|
||||||
|
login position). File-name-only, so no Xaero format knowledge needed; reports before deleting.
|
||||||
|
|
||||||
|
```
|
||||||
|
tools/filter-xaero-regions.py <game>/xaero/world-map/Multiplayer_advilplug.top/null/ mask.json
|
||||||
|
```
|
||||||
|
|
||||||
|
`tools/prune-world-to-reveal.py` - **validated end-to-end against a real region file from the server**
|
||||||
|
(`world/region/r.1.-2.mca`, 15.7 MB):
|
||||||
|
|
||||||
|
| Check | Result |
|
||||||
|
| --- | --- |
|
||||||
|
| Anvil parse (1024 slots, all zlib, 2.5-21.8 KB payloads) | OK |
|
||||||
|
| Slot -> chunk mapping (`slot = x + z*32`), confirmed by reading `xPos`/`zPos` NBT | **1024 / 1024 exact** |
|
||||||
|
| Prune against that player's FTB mask (52 of 1024 chunks revealed) | 52 kept / 972 dropped |
|
||||||
|
| Output sector-aligned | yes |
|
||||||
|
| Kept slot set == mask | yes |
|
||||||
|
| Kept payloads + timestamps byte-identical | yes |
|
||||||
|
| Kept chunks re-parse with correct coordinates | 52 / 52 |
|
||||||
|
| Size | 15,745,024 -> 471,040 bytes (3.0%) |
|
||||||
|
|
||||||
|
Earlier synthetic tests also cover nested per-dimension `region`/`entities`/`poi` trees not being
|
||||||
|
duplicated.
|
||||||
|
|
||||||
|
```
|
||||||
|
tools/prune-world-to-reveal.py <world-copy> mask.json <out-dir> --dimension minecraft:overworld
|
||||||
|
```
|
||||||
|
|
||||||
|
`--minimal` writes only `level.dat` plus the pruned `region/` tree - the smallest thing Xaero's can
|
||||||
|
open and map, and the best fit when the full save is too large to copy around. `--dry-run` reports
|
||||||
|
kept/dropped counts per region without writing anything. Runtime is ~0.15 s per 15 MB region.
|
||||||
|
|
||||||
|
## 6. Running it where the save lives
|
||||||
|
|
||||||
|
The tools are stdlib-only Python, so nothing has to be transferred: run them on the machine that holds
|
||||||
|
the world. The live world is **not** on the VPS (the VPS is only a proxy in front of the real server),
|
||||||
|
so this has to run where the save actually lives. The per-player footprint is only the pruned `region/`
|
||||||
|
tree for the regions that player revealed, not the whole save.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
tools/migrate-player.sh Stirwick <stirwick-ftbchunks-data> <world-dir> \
|
||||||
|
--dimension minecraft:overworld --work /var/tmp/xaero-migration
|
||||||
|
```
|
||||||
|
|
||||||
|
That extracts the mask (with the `chunks.dat` vs `data.png` cross-check) and prunes the world,
|
||||||
|
dry-run first. Then, by hand:
|
||||||
|
|
||||||
|
3. Copy the pruned world (`<work>/<player>/sp-world`) to a machine that can run the pack and open it in
|
||||||
|
**singleplayer** once. Let Xaero's map it, then quit.
|
||||||
|
4. Move the generated instance, `xaero/world-map/World/null/<map>/`, into the server's map folder,
|
||||||
|
`xaero/world-map/Multiplayer_advilplug.top/<DIM>/`.
|
||||||
|
5. Run `tools/filter-xaero-regions.py` on that folder to drop regions the player never revealed.
|
||||||
|
6. Delete the scratch directory - usually a few hundred MB.
|
||||||
|
|
||||||
|
### Two things that will bite
|
||||||
|
|
||||||
|
- **The server does not need stopping, but freeze saves while it runs.** The pruner only reads the world
|
||||||
|
and writes to a separate output directory. However a running server rewrites region files on autosave
|
||||||
|
and rewrites `level.dat` in place, so a read can tear - and half a `level.dat` is an unopenable
|
||||||
|
singleplayer world. From the console: `save-off` -> dry run -> real run -> `save-all` -> `save-on`.
|
||||||
|
The pass takes a couple of minutes. Never forget `save-on`, and never point `--out` at the live world.
|
||||||
|
A region that does fail to read is reported, skipped, and left unwritten; re-run to retry it.
|
||||||
|
|
||||||
|
- **Opening the world regenerates chunks near the login position.** Minecraft loads and generates the
|
||||||
|
chunks around wherever the player spawns, so terrain the player never revealed can reappear on the
|
||||||
|
map if it happens to sit near world spawn. It is bounded by render distance (well under one
|
||||||
|
512x512 region at typical settings), and it is easy to undo: step 4's cleanup compares the harvested
|
||||||
|
region file names against the mask and deletes any region the player never revealed. Keeping
|
||||||
|
`level.dat`'s spawn inside the revealed area, or not walking around in step 3, avoids the rest.
|
||||||
|
- **Do not copy the world with hardlinks.** `cp -al` would make the singleplayer client's own writes to
|
||||||
|
`level.dat` land in the live server save. A real copy, or `--minimal` into a fresh directory, is safe.
|
||||||
|
|
||||||
|
## 7. Open items
|
||||||
|
|
||||||
|
- **Needs: the full world save, read in place on the server.** One real region file (`r.1.-2.mca`) has
|
||||||
|
now validated the reader, the slot mapping and the writer, so the remaining unknowns are operational
|
||||||
|
rather than format-related: total runtime over 167 regions (extrapolating ~0.15 s per 15 MB region,
|
||||||
|
a couple of minutes) and the size of the pruned output. A full 1024-chunk region on this server is
|
||||||
|
~15 MB; the player revealed 57.4% of the bounding box overall, so expect the pruned per-player world
|
||||||
|
to land in the low hundreds of MB. It is scratch data and can be deleted once the Xaero's map has
|
||||||
|
been harvested.
|
||||||
|
- **Per-player, per-dimension.** Only `minecraft:overworld` was supplied and only one player's data.
|
||||||
|
Modded dimensions will each need an FTB-data -> Xaero-dimension mapping. Xaero names its dimension
|
||||||
|
folders `null`/`DIM-1`/`DIM1` for vanilla and uses the `minecraft$overworld` form (`$`, not `:`) in
|
||||||
|
`dimension_config.txt`; a `dimensions.txt` with more than one entry will confirm the rest.
|
||||||
|
- **Step 3 costs one client launch per player.** It cannot be done server-side; Xaero's maps from a save
|
||||||
|
on the client. ~10 launches for this pack.
|
||||||
|
- **Waypoints are a separate, easier win.** FTB stores them in `waypoints.json` next to the region data
|
||||||
|
(empty, 16 bytes, for the sampled player). Xaero waypoints are plain files under `xaero/minimap/`,
|
||||||
|
so a converter is straightforward - worth doing at the same time if anyone has waypoints to keep.
|
||||||
|
- **Fallbacks if the save never materialises:** region-granularity pruning (coarse, ~2x over-reveal), or
|
||||||
|
shipping the full "Convert All Regions" map to everyone (over-reveals everything, zero code).
|
||||||
|
- Nothing here is time-sensitive: removing the mod from the pack does not touch `local/ftbchunks/`, so
|
||||||
|
every client's reveal data is still on disk and this remains doable later.
|
||||||
Executable
+88
@@ -0,0 +1,88 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Drop Xaero's map regions that a player never actually revealed.
|
||||||
|
|
||||||
|
After a player opens their pruned singleplayer world, Xaero's maps it - but Minecraft
|
||||||
|
also *regenerates* the chunks around wherever that player logged in, so a little
|
||||||
|
terrain they had never seen can sneak onto the map. This compares the harvested map
|
||||||
|
against the reveal mask and removes the strays.
|
||||||
|
|
||||||
|
It works on file names only (`<rx>_<rz>.zip` is one 512x512-block region, exactly the
|
||||||
|
same geometry as an FTB Chunks region file), so it needs no knowledge of Xaero's
|
||||||
|
region format.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
filter-xaero-regions.py <xaero-map-dir> <mask.json> [--dimension <id>] [--delete]
|
||||||
|
|
||||||
|
Without --delete it only reports. Run it on one dimension folder at a time, e.g.
|
||||||
|
~/.minecraft/xaero/world-map/Multiplayer_advilplug.top/null/
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
|
||||||
|
REGION_RE = re.compile(r"^(-?\d+)_(-?\d+)\.zip$")
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
|
||||||
|
ap.add_argument("map_dir", help="Xaero's map instance directory holding <rx>_<rz>.zip")
|
||||||
|
ap.add_argument("mask", help="mask JSON from ftbchunks-reveal.py")
|
||||||
|
ap.add_argument("--dimension", default=None,
|
||||||
|
help="dimension to check (default: the mask's only dimension)")
|
||||||
|
ap.add_argument("--delete", action="store_true", help="actually remove the strays")
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
with open(args.mask) as fh:
|
||||||
|
mask = json.load(fh)
|
||||||
|
dims = mask["dimensions"]
|
||||||
|
dim = args.dimension
|
||||||
|
if dim is None:
|
||||||
|
if len(dims) != 1:
|
||||||
|
print(f"mask has {len(dims)} dimensions; pass --dimension", file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
dim = next(iter(dims))
|
||||||
|
if dim not in dims:
|
||||||
|
print(f"no dimension '{dim}' in mask; have {list(dims)}", file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
|
||||||
|
revealed = {tuple(int(v) for v in key.split(",")) for key in dims[dim]["regions"]}
|
||||||
|
|
||||||
|
if not os.path.isdir(args.map_dir):
|
||||||
|
print(f"not a directory: {args.map_dir}", file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
|
||||||
|
present, strays = [], []
|
||||||
|
for name in sorted(os.listdir(args.map_dir)):
|
||||||
|
m = REGION_RE.match(name)
|
||||||
|
if not m:
|
||||||
|
continue
|
||||||
|
coords = (int(m.group(1)), int(m.group(2)))
|
||||||
|
(present if coords in revealed else strays).append((name, coords))
|
||||||
|
|
||||||
|
print(f"dimension {dim}")
|
||||||
|
print(f" map regions present : {len(present)} (of {len(revealed)} revealed)")
|
||||||
|
print(f" not in the mask : {len(strays)}")
|
||||||
|
for name, _ in strays:
|
||||||
|
print(f" {name}")
|
||||||
|
|
||||||
|
missing = sorted(revealed - {c for _, c in present})
|
||||||
|
if missing:
|
||||||
|
print(f" revealed but absent from the map: {len(missing)} "
|
||||||
|
f"(e.g. {missing[:4]}) - regenerate these, do not delete anything")
|
||||||
|
|
||||||
|
if args.delete and strays:
|
||||||
|
for name, _ in strays:
|
||||||
|
os.remove(os.path.join(args.map_dir, name))
|
||||||
|
print(f"\nremoved {len(strays)} stray region file(s)")
|
||||||
|
elif strays:
|
||||||
|
print("\nre-run with --delete to remove them")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
Executable
+284
@@ -0,0 +1,284 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Extract a per-chunk reveal mask from FTB Chunks client map data.
|
||||||
|
|
||||||
|
FTB Chunks stores its explored-terrain map client-side, one zip per 512x512-block
|
||||||
|
region:
|
||||||
|
|
||||||
|
<game>/local/ftbchunks/data/<serverUUID>/<dimension>/<XXXXX-XXXXX>.zip
|
||||||
|
|
||||||
|
Each zip holds `chunks.dat` (the chunks the player has actually seen) plus five
|
||||||
|
PNG data images. Region file names are the region coordinate + 60000, formatted
|
||||||
|
`%05X-%05X`, which is why overworld spawn shows up as `0EA60-0EA60.zip`.
|
||||||
|
|
||||||
|
`chunks.dat` layout (verified against ftb-chunks-neoforge-2101.1.22 bytecode):
|
||||||
|
|
||||||
|
u8 leading byte (0)
|
||||||
|
u8 version (3 on this build)
|
||||||
|
i16 big-endian chunk count
|
||||||
|
per chunk:
|
||||||
|
u8 chunk version (present when version >= 2)
|
||||||
|
u8 x within region (0..31)
|
||||||
|
u8 z within region (0..31)
|
||||||
|
i64 big-endian last-modified millis
|
||||||
|
|
||||||
|
`chunks.dat` was checked against the `data.png` height sentinel (HeightUtils.UNKNOWN
|
||||||
|
= -32767, packed as (height << 16) | (water|light|biome)) across all 167 regions of a
|
||||||
|
real 98,170-chunk dataset: zero disagreements. So chunks.dat alone is a trustworthy
|
||||||
|
reveal mask; --verify re-runs that cross-check anyway.
|
||||||
|
|
||||||
|
Output is a JSON mask consumed by prune-world-to-reveal.py.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
ftbchunks-reveal.py <input> [--out mask.json] [--dimension <id>] [--verify] [--quiet]
|
||||||
|
|
||||||
|
<input> is either
|
||||||
|
* a client data directory, e.g. .../local/ftbchunks/data/<uuid>/, which is scanned
|
||||||
|
across all dimensions listed in its dimensions.txt, or
|
||||||
|
* a directory or zip containing region zips directly (a single dimension, as when
|
||||||
|
someone has archived just the overworld). Such an input cannot name its own
|
||||||
|
dimension, so pass --dimension to label it.
|
||||||
|
|
||||||
|
Output coordinates are global chunk coordinates, so they are independent of which
|
||||||
|
region file a chunk happened to live in.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import io
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import struct
|
||||||
|
import sys
|
||||||
|
import zipfile
|
||||||
|
import zlib
|
||||||
|
|
||||||
|
REGION_RE = re.compile(r"^([0-9A-Fa-f]{4,6})-([0-9A-Fa-f]{4,6})\.zip$")
|
||||||
|
REGION_OFFSET = 60000
|
||||||
|
|
||||||
|
# HeightUtils.UNKNOWN = Short.MIN_VALUE + 1, stored masked to 16 bits.
|
||||||
|
UNKNOWN_RAW = (-32767) & 0xFFFF
|
||||||
|
|
||||||
|
|
||||||
|
def parse_region_name(name: str) -> tuple[int, int] | None:
|
||||||
|
"""`0EA60-0EA5F.zip` -> region coordinate (0, -1)."""
|
||||||
|
m = REGION_RE.match(os.path.basename(name))
|
||||||
|
if not m:
|
||||||
|
return None
|
||||||
|
return int(m.group(1), 16) - REGION_OFFSET, int(m.group(2), 16) - REGION_OFFSET
|
||||||
|
|
||||||
|
|
||||||
|
def parse_chunks_dat(data: bytes) -> set[tuple[int, int]]:
|
||||||
|
"""Return the within-region (x, z) chunk coordinates present in chunks.dat."""
|
||||||
|
if len(data) < 4:
|
||||||
|
return set()
|
||||||
|
buf = io.BytesIO(data)
|
||||||
|
buf.read(1)
|
||||||
|
version = buf.read(1)[0]
|
||||||
|
count = struct.unpack(">h", buf.read(2))[0]
|
||||||
|
chunks: set[tuple[int, int]] = set()
|
||||||
|
for _ in range(count):
|
||||||
|
if version >= 2:
|
||||||
|
buf.read(1)
|
||||||
|
x = buf.read(1)[0]
|
||||||
|
z = buf.read(1)[0]
|
||||||
|
buf.read(8)
|
||||||
|
chunks.add((x, z))
|
||||||
|
return chunks
|
||||||
|
|
||||||
|
|
||||||
|
# --- optional cross-check: decode data.png and find the UNKNOWN sentinel --------
|
||||||
|
|
||||||
|
def _png_decode(data: bytes) -> tuple[int, int, int, bytes]:
|
||||||
|
"""Minimal PNG decoder (8-bit, non-interlaced) -> (w, h, channels, rgba bytes)."""
|
||||||
|
if data[:8] != b"\x89PNG\r\n\x1a\n":
|
||||||
|
raise ValueError("not a PNG")
|
||||||
|
pos, idat = 8, b""
|
||||||
|
width = height = depth = colour = None
|
||||||
|
while pos < len(data):
|
||||||
|
(length,) = struct.unpack(">I", data[pos : pos + 4])
|
||||||
|
kind = data[pos + 4 : pos + 8]
|
||||||
|
body = data[pos + 8 : pos + 8 + length]
|
||||||
|
pos += 12 + length
|
||||||
|
if kind == b"IHDR":
|
||||||
|
width, height, depth, colour, comp, filt, interlace = struct.unpack(
|
||||||
|
">IIBBBBB", body
|
||||||
|
)
|
||||||
|
if depth != 8 or interlace != 0:
|
||||||
|
raise ValueError(f"unsupported PNG depth/interlace {depth}/{interlace}")
|
||||||
|
elif kind == b"IDAT":
|
||||||
|
idat += body
|
||||||
|
elif kind == b"IEND":
|
||||||
|
break
|
||||||
|
channels = {0: 1, 2: 3, 3: 1, 4: 2, 6: 4}[colour]
|
||||||
|
stride = width * channels
|
||||||
|
raw = zlib.decompress(idat)
|
||||||
|
out = bytearray()
|
||||||
|
prev = bytearray(stride)
|
||||||
|
p = 0
|
||||||
|
for _ in range(height):
|
||||||
|
ftype = raw[p]
|
||||||
|
p += 1
|
||||||
|
line = bytearray(raw[p : p + stride])
|
||||||
|
p += stride
|
||||||
|
if ftype == 1:
|
||||||
|
for i in range(channels, stride):
|
||||||
|
line[i] = (line[i] + line[i - channels]) & 0xFF
|
||||||
|
elif ftype == 2:
|
||||||
|
for i in range(stride):
|
||||||
|
line[i] = (line[i] + prev[i]) & 0xFF
|
||||||
|
elif ftype == 3:
|
||||||
|
for i in range(stride):
|
||||||
|
a = line[i - channels] if i >= channels else 0
|
||||||
|
line[i] = (line[i] + ((a + prev[i]) >> 1)) & 0xFF
|
||||||
|
elif ftype == 4:
|
||||||
|
for i in range(stride):
|
||||||
|
a = line[i - channels] if i >= channels else 0
|
||||||
|
b = prev[i]
|
||||||
|
c = prev[i - channels] if i >= channels else 0
|
||||||
|
pp = a + b - c
|
||||||
|
pa, pb, pc = abs(pp - a), abs(pp - b), abs(pp - c)
|
||||||
|
pred = a if (pa <= pb and pa <= pc) else (b if pb <= pc else c)
|
||||||
|
line[i] = (line[i] + pred) & 0xFF
|
||||||
|
out += line
|
||||||
|
prev = line
|
||||||
|
return width, height, channels, bytes(out)
|
||||||
|
|
||||||
|
|
||||||
|
def chunks_seen_in_png(png_bytes: bytes) -> set[tuple[int, int]]:
|
||||||
|
"""Chunks (0..31)^2 having at least one non-UNKNOWN pixel."""
|
||||||
|
width, height, channels, px = _png_decode(png_bytes)
|
||||||
|
unknown_r = UNKNOWN_RAW & 0xFF
|
||||||
|
unknown_a = (UNKNOWN_RAW >> 8) & 0xFF
|
||||||
|
seen: set[tuple[int, int]] = set()
|
||||||
|
for cz in range(32):
|
||||||
|
for cx in range(32):
|
||||||
|
hit = False
|
||||||
|
for yy in range(16):
|
||||||
|
base = ((cz * 16 + yy) * width + cx * 16) * channels
|
||||||
|
for xx in range(16):
|
||||||
|
i = base + xx * channels
|
||||||
|
if not (px[i] == unknown_r and px[i + 3] == unknown_a):
|
||||||
|
hit = True
|
||||||
|
break
|
||||||
|
if hit:
|
||||||
|
break
|
||||||
|
if hit:
|
||||||
|
seen.add((cx, cz))
|
||||||
|
return seen
|
||||||
|
|
||||||
|
|
||||||
|
# --- input discovery -----------------------------------------------------------
|
||||||
|
|
||||||
|
def iter_region_zips(path: str):
|
||||||
|
"""Yield (dimension_name, region_filename, bytes) from a dir or an archive."""
|
||||||
|
if zipfile.is_zipfile(path):
|
||||||
|
with zipfile.ZipFile(path) as outer:
|
||||||
|
for info in outer.infolist():
|
||||||
|
if parse_region_name(info.filename) is None:
|
||||||
|
continue
|
||||||
|
yield "(unspecified)", os.path.basename(info.filename), outer.read(info)
|
||||||
|
return
|
||||||
|
if os.path.isdir(path) and any(
|
||||||
|
parse_region_name(n) for n in os.listdir(path)
|
||||||
|
):
|
||||||
|
for name in sorted(os.listdir(path)):
|
||||||
|
if parse_region_name(name):
|
||||||
|
with open(os.path.join(path, name), "rb") as fh:
|
||||||
|
yield "(unspecified)", name, fh.read()
|
||||||
|
return
|
||||||
|
# treat as a client data directory keyed by dimension
|
||||||
|
dims_file = os.path.join(path, "dimensions.txt")
|
||||||
|
dims = []
|
||||||
|
if os.path.exists(dims_file):
|
||||||
|
with open(dims_file) as fh:
|
||||||
|
dims = [ln.strip() for ln in fh if ln.strip()]
|
||||||
|
else:
|
||||||
|
dims = [
|
||||||
|
d
|
||||||
|
for d in sorted(os.listdir(path))
|
||||||
|
if os.path.isdir(os.path.join(path, d))
|
||||||
|
]
|
||||||
|
for dim in dims:
|
||||||
|
dim_dir = os.path.join(path, dim)
|
||||||
|
if not os.path.isdir(dim_dir):
|
||||||
|
continue
|
||||||
|
for name in sorted(os.listdir(dim_dir)):
|
||||||
|
if parse_region_name(name):
|
||||||
|
with open(os.path.join(dim_dir, name), "rb") as fh:
|
||||||
|
yield dim, name, fh.read()
|
||||||
|
|
||||||
|
|
||||||
|
def extract(path: str, verify: bool, quiet: bool, dimension_label: str | None = None) -> dict:
|
||||||
|
dims: dict[str, dict] = {}
|
||||||
|
total = 0
|
||||||
|
mismatches = 0
|
||||||
|
for dim, name, blob in iter_region_zips(path):
|
||||||
|
if dim == "(unspecified)" and dimension_label:
|
||||||
|
dim = dimension_label
|
||||||
|
coords = parse_region_name(name)
|
||||||
|
if coords is None:
|
||||||
|
continue
|
||||||
|
rx, rz = coords
|
||||||
|
with zipfile.ZipFile(io.BytesIO(blob)) as region:
|
||||||
|
listed = parse_chunks_dat(region.read("chunks.dat"))
|
||||||
|
if verify:
|
||||||
|
seen = chunks_seen_in_png(region.read("data.png"))
|
||||||
|
if seen != listed:
|
||||||
|
mismatches += 1
|
||||||
|
print(
|
||||||
|
f" ! {dim}/{name}: chunks.dat={len(listed)} png={len(seen)}",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
entry = dims.setdefault(dim, {"regions": {}, "chunks": []})
|
||||||
|
entry["regions"][f"{rx},{rz}"] = len(listed)
|
||||||
|
for cx, cz in sorted(listed):
|
||||||
|
entry["chunks"].append([rx * 32 + cx, rz * 32 + cz])
|
||||||
|
total += len(listed)
|
||||||
|
if not quiet:
|
||||||
|
print(f" {dim:34} {name:>16} {len(listed):>4} chunks", file=sys.stderr)
|
||||||
|
|
||||||
|
for entry in dims.values():
|
||||||
|
entry["chunks"].sort()
|
||||||
|
entry["region_count"] = len(entry["regions"])
|
||||||
|
entry["chunk_count"] = len(entry["chunks"])
|
||||||
|
return {
|
||||||
|
"source": os.path.abspath(path),
|
||||||
|
"verified": verify,
|
||||||
|
"mismatches": mismatches,
|
||||||
|
"total_chunks": total,
|
||||||
|
"dimensions": dims,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
|
||||||
|
ap.add_argument("input", help="FTB Chunks data dir, or a dir/zip of region zips")
|
||||||
|
ap.add_argument("--out", default="ftb-reveal-mask.json")
|
||||||
|
ap.add_argument("--dimension", default=None,
|
||||||
|
help="dimension id used to label a bare directory/zip of region files, which "
|
||||||
|
"cannot say which dimension it is (e.g. minecraft:overworld)")
|
||||||
|
ap.add_argument("--verify", action="store_true", help="cross-check chunks.dat vs data.png")
|
||||||
|
ap.add_argument("--quiet", action="store_true")
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
mask = extract(args.input, args.verify, args.quiet, args.dimension)
|
||||||
|
with open(args.out, "w") as fh:
|
||||||
|
json.dump(mask, fh)
|
||||||
|
|
||||||
|
if args.verify:
|
||||||
|
status = "OK" if mask["mismatches"] == 0 else f"{mask['mismatches']} MISMATCHES"
|
||||||
|
print(f"cross-check chunks.dat vs data.png: {status}", file=sys.stderr)
|
||||||
|
print(
|
||||||
|
f"{mask['total_chunks']:,} revealed chunks in "
|
||||||
|
f"{sum(d['region_count'] for d in mask['dimensions'].values())} regions "
|
||||||
|
f"across {len(mask['dimensions'])} dimension(s) -> {args.out}"
|
||||||
|
)
|
||||||
|
for dim, d in mask["dimensions"].items():
|
||||||
|
print(f" {dim}: {d['chunk_count']:,} chunks / {d['region_count']} regions")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
Executable
+131
@@ -0,0 +1,131 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# End-to-end driver for the FTB Chunks -> Xaero's World Map reveal transplant.
|
||||||
|
#
|
||||||
|
# See analysis/FTB-XAERO-MAP-MIGRATION.md for the reasoning. This script does the
|
||||||
|
# parts that can be automated; the singleplayer pass in step 3 has to be done by hand
|
||||||
|
# on a machine that can run the pack.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# migrate-player.sh <player> <ftbchunks-data> <world-dir> [--dimension <id>] [--work <dir>]
|
||||||
|
#
|
||||||
|
# <player> label used for output paths, e.g. Stirwick
|
||||||
|
# <ftbchunks-data> that player's local/ftbchunks/data/<uuid>/, or a bare zip/dir
|
||||||
|
# of region files, or an already-extracted mask .json
|
||||||
|
# <world-dir> the world save containing region/
|
||||||
|
#
|
||||||
|
# Example:
|
||||||
|
# tools/migrate-player.sh Stirwick ~/Stirwick-ftbchunks ~/server/world \
|
||||||
|
# --dimension minecraft:overworld --work /var/tmp/xaero-migration
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
|
||||||
|
if [ $# -lt 3 ]; then
|
||||||
|
sed -n '2,20p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
|
||||||
|
player="$1"; ftb="$2"; world="$3"; shift 3
|
||||||
|
dimension="minecraft:overworld"
|
||||||
|
work="$(pwd)/xaero-migration"
|
||||||
|
|
||||||
|
while [ $# -gt 0 ]; do
|
||||||
|
case "$1" in
|
||||||
|
--dimension) dimension="$2"; shift 2 ;;
|
||||||
|
--work) work="$2"; shift 2 ;;
|
||||||
|
*) echo "unknown option: $1" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
out="$work/$player"
|
||||||
|
mask="$out/mask.json"
|
||||||
|
sp="$out/sp-world"
|
||||||
|
|
||||||
|
echo "player : $player"
|
||||||
|
echo "ftb data : $ftb"
|
||||||
|
echo "world : $world"
|
||||||
|
echo "dimension : $dimension"
|
||||||
|
echo "output : $out"
|
||||||
|
echo
|
||||||
|
|
||||||
|
if [ ! -d "$world/region" ]; then
|
||||||
|
echo "error: $world/region not found" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
mkdir -p "$out"
|
||||||
|
|
||||||
|
echo "== 1/2 extracting reveal mask =="
|
||||||
|
if [ "${ftb##*.}" = "json" ]; then
|
||||||
|
cp -f "$ftb" "$mask"
|
||||||
|
echo " using prebuilt mask: $ftb"
|
||||||
|
python3 - "$mask" <<'PY'
|
||||||
|
import json, sys
|
||||||
|
m = json.load(open(sys.argv[1]))
|
||||||
|
dims = m["dimensions"]
|
||||||
|
print(f" {m['total_chunks']:,} chunks / "
|
||||||
|
f"{sum(d['region_count'] for d in dims.values())} regions / "
|
||||||
|
f"dimension(s) {list(dims)}")
|
||||||
|
PY
|
||||||
|
else
|
||||||
|
python3 "$here/ftbchunks-reveal.py" "$ftb" --out "$mask" --dimension "$dimension" --verify
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "== 2/2 pruning the world to those chunks (dry run first) =="
|
||||||
|
python3 "$here/prune-world-to-reveal.py" "$world" "$mask" "$sp" \
|
||||||
|
--dimension "$dimension" --minimal --dry-run
|
||||||
|
echo
|
||||||
|
python3 "$here/prune-world-to-reveal.py" "$world" "$mask" "$sp" \
|
||||||
|
--dimension "$dimension" --minimal
|
||||||
|
|
||||||
|
cat <<EOF
|
||||||
|
|
||||||
|
Done. Pruned world: $sp
|
||||||
|
|
||||||
|
Now, by hand.
|
||||||
|
|
||||||
|
3. Copy $sp to a machine that can run the pack and open it in SINGLEPLAYER once,
|
||||||
|
as a world folder (the directory holding level.dat and region/):
|
||||||
|
|
||||||
|
<game dir>/saves/Stirwick-reveal/
|
||||||
|
|
||||||
|
Open it, open the map (M), let it fill, then quit. Xaero's maps what is in the
|
||||||
|
save, and this save holds only the revealed chunks. Do not walk around --
|
||||||
|
Minecraft regenerates chunks near the login position and they will show up.
|
||||||
|
|
||||||
|
Where the result lands: look under <game dir>/xaero/world-map/ for a folder that
|
||||||
|
is not Multiplayer_*, e.g. World/null/<map>/. Sort by modification time if
|
||||||
|
unsure -- those files were just written.
|
||||||
|
|
||||||
|
SANITY CHECK: you should end up with one zip per region in the mask
|
||||||
|
($(python3 -c "import json;print(sum(d['region_count'] for d in json.load(open('$mask'))['dimensions'].values()))") for $player). A handful means it only mapped the spawn area and
|
||||||
|
you should use 'Convert All Regions' in the world map settings, or report back.
|
||||||
|
|
||||||
|
4. Clean the HARVEST before anything else -- this is the step that removes terrain
|
||||||
|
Minecraft regenerated around the login point. Run it on the harvested folder,
|
||||||
|
NOT on the player's own map folder:
|
||||||
|
|
||||||
|
python3 $here/filter-xaero-regions.py \\
|
||||||
|
"<game dir>/xaero/world-map/World/null/<map>/" \\
|
||||||
|
$mask --dimension $dimension
|
||||||
|
|
||||||
|
Check the list, then re-run with --delete.
|
||||||
|
|
||||||
|
5. Merge the surviving <rx>_<rz>.zip files into the player's existing map folder,
|
||||||
|
WITHOUT overwriting anything already there (-n), so any terrain they have
|
||||||
|
explored since the swap is kept:
|
||||||
|
|
||||||
|
cp -n <harvested>/*.zip \\
|
||||||
|
"<their game dir>/xaero/world-map/Multiplayer_advilplug.top/null/<their map instance>/"
|
||||||
|
|
||||||
|
The right instance is the mw\$... folder there. If the player has not joined the
|
||||||
|
server since the swap, have them join once first so Xaero's creates it.
|
||||||
|
|
||||||
|
6. Delete $out -- scratch, usually a few hundred MB.
|
||||||
|
|
||||||
|
Per-dimension: repeat with --dimension <id> and the matching map folder
|
||||||
|
(null = overworld, DIM-1 = nether, DIM1 = end).
|
||||||
|
Other players: each needs their own local/ftbchunks/data/<uuid>/; there is no
|
||||||
|
server-side copy of this data.
|
||||||
|
EOF
|
||||||
Executable
+210
@@ -0,0 +1,210 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Prune a Minecraft Anvil world save down to a set of revealed chunks.
|
||||||
|
|
||||||
|
This is how we transplant FTB Chunks' reveal footprint onto Xaero's World Map
|
||||||
|
without reverse-engineering Xaero's region format: strip the world save down to
|
||||||
|
exactly the chunks a player had explored, then let Xaero's own generator map that
|
||||||
|
save in singleplayer. Xaero's renders it natively (correct palette, shading,
|
||||||
|
light, biome) and the absent chunks simply never appear on the map.
|
||||||
|
|
||||||
|
Chunk payloads are copied through byte-for-byte -- no NBT re-encoding -- so the
|
||||||
|
output is safe for modded 1.21.1 chunk data. Only the 8 KiB header, the sector
|
||||||
|
offsets and the zero padding are rewritten.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
prune-world-to-reveal.py <world-save-dir> <mask.json> <out-dir>
|
||||||
|
[--dimension minecraft:overworld] [--dry-run]
|
||||||
|
|
||||||
|
Reads `<world-save-dir>/region/r.<x>.<z>.mca` and writes pruned copies to
|
||||||
|
`<out-dir>/region/`, along with whatever else the world needs. Nested per-dimension
|
||||||
|
`region`/`entities`/`poi` trees are never copied, so the Nether and End are not
|
||||||
|
duplicated at full size; `--minimal` goes further and copies only `level.dat`, which
|
||||||
|
is all Xaero's needs to map the overworld.
|
||||||
|
|
||||||
|
If a full save copy is wanted (level.dat, data/, playerdata/ ...), copy the world
|
||||||
|
first and then run this in place with <out-dir> == <world-save-dir>.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import shutil
|
||||||
|
import struct
|
||||||
|
import sys
|
||||||
|
|
||||||
|
SECTOR = 4096
|
||||||
|
HEADER = 2 * SECTOR
|
||||||
|
COMPRESSION_NAMES = {1: "gzip", 2: "zlib", 3: "none", 4: "lz4", 127: "external"}
|
||||||
|
|
||||||
|
|
||||||
|
def read_slot(data: bytes, size: int, slot: int) -> tuple[bytes, int, int] | None:
|
||||||
|
"""Return (payload, timestamp, compression) for an Anvil slot, or None if empty."""
|
||||||
|
off = struct.unpack(">I", b"\x00" + data[slot * 4 : slot * 4 + 3])[0]
|
||||||
|
if off == 0:
|
||||||
|
return None
|
||||||
|
timestamp = struct.unpack(">I", data[SECTOR + slot * 4 : SECTOR + slot * 4 + 4])[0]
|
||||||
|
start = off * SECTOR
|
||||||
|
if start + 5 > len(data):
|
||||||
|
raise ValueError(f"slot {slot}: offset {off} past end of file")
|
||||||
|
(length,) = struct.unpack(">I", data[start : start + 4])
|
||||||
|
compression = data[start + 4]
|
||||||
|
if length < 1 or start + 4 + length > len(data):
|
||||||
|
raise ValueError(f"slot {slot}: bad chunk length {length}")
|
||||||
|
return data[start + 5 : start + 4 + length], timestamp, compression
|
||||||
|
|
||||||
|
|
||||||
|
def prune_region(path: str, keep_slots: set[int], out_path: str, dry: bool) -> dict:
|
||||||
|
with open(path, "rb") as fh:
|
||||||
|
data = fh.read()
|
||||||
|
|
||||||
|
kept: list[tuple[int, bytes, int, int]] = []
|
||||||
|
dropped = 0
|
||||||
|
external = 0
|
||||||
|
for slot in range(1024):
|
||||||
|
entry = read_slot(data, SECTOR, slot)
|
||||||
|
if entry is None:
|
||||||
|
continue
|
||||||
|
payload, timestamp, compression = entry
|
||||||
|
if compression == 127:
|
||||||
|
external += 1
|
||||||
|
if slot in keep_slots:
|
||||||
|
kept.append((slot, payload, timestamp, compression))
|
||||||
|
else:
|
||||||
|
dropped += 1
|
||||||
|
|
||||||
|
stats = {
|
||||||
|
"file": os.path.basename(path),
|
||||||
|
"kept": len(kept),
|
||||||
|
"dropped": dropped,
|
||||||
|
"external_payloads": external,
|
||||||
|
}
|
||||||
|
if dry:
|
||||||
|
return stats
|
||||||
|
|
||||||
|
out = bytearray(HEADER)
|
||||||
|
sector = 2
|
||||||
|
for slot, payload, timestamp, compression in kept:
|
||||||
|
blob = struct.pack(">I", len(payload) + 1) + bytes([compression]) + payload
|
||||||
|
sectors = (len(blob) + SECTOR - 1) // SECTOR
|
||||||
|
if sector + sectors > 0xFFFFFF:
|
||||||
|
raise ValueError(f"{path}: region would exceed the Anvil 24-bit offset limit")
|
||||||
|
out[slot * 4 : slot * 4 + 3] = sector.to_bytes(3, "big")
|
||||||
|
out[slot * 4 + 3] = sectors
|
||||||
|
out[SECTOR + slot * 4 : SECTOR + slot * 4 + 4] = struct.pack(">I", timestamp)
|
||||||
|
out += blob
|
||||||
|
out += b"\x00" * (sectors * SECTOR - len(blob))
|
||||||
|
sector += sectors
|
||||||
|
|
||||||
|
os.makedirs(os.path.dirname(out_path) or ".", exist_ok=True)
|
||||||
|
with open(out_path, "wb") as fh:
|
||||||
|
fh.write(out)
|
||||||
|
return stats
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
|
||||||
|
ap.add_argument("world", help="world save directory containing region/")
|
||||||
|
ap.add_argument("mask", help="mask JSON from ftbchunks-reveal.py")
|
||||||
|
ap.add_argument("out", help="output world directory (use the same path for in-place)")
|
||||||
|
ap.add_argument("--dimension", default="minecraft:overworld",
|
||||||
|
help="which dimension in the mask to apply (default overworld)")
|
||||||
|
ap.add_argument("--minimal", action="store_true",
|
||||||
|
help="output only level.dat + the pruned region/ tree (smallest usable world)")
|
||||||
|
ap.add_argument("--dry-run", action="store_true")
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
with open(args.mask) as fh:
|
||||||
|
mask = json.load(fh)
|
||||||
|
dims = mask["dimensions"]
|
||||||
|
if args.dimension not in dims:
|
||||||
|
print(f"mask has no dimension '{args.dimension}'; available: {list(dims)}", file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
|
||||||
|
chunks = {(cx, cz) for cx, cz in dims[args.dimension]["chunks"]}
|
||||||
|
by_region: dict[tuple[int, int], set[int]] = {}
|
||||||
|
for cx, cz in chunks:
|
||||||
|
rx, rz = cx >> 5, cz >> 5
|
||||||
|
by_region.setdefault((rx, rz), set()).add((cx & 31) + (cz & 31) * 32)
|
||||||
|
|
||||||
|
in_region = os.path.join(args.world, "region")
|
||||||
|
if not os.path.isdir(in_region):
|
||||||
|
print(f"no region/ directory under {args.world}", file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
out_region = os.path.join(args.out, "region")
|
||||||
|
os.makedirs(out_region, exist_ok=True)
|
||||||
|
|
||||||
|
# Copy the rest of the world so the pruned save is openable, unless in-place.
|
||||||
|
# Nested `region`/`entities`/`poi` trees (i.e. per-dimension ones like DIM-1/) are never
|
||||||
|
# copied - copying them wholesale would duplicate the Nether and End at full size.
|
||||||
|
if os.path.abspath(args.out) != os.path.abspath(args.world) and not args.dry_run:
|
||||||
|
ignore = shutil.ignore_patterns("region", "entities", "poi")
|
||||||
|
for name in sorted(os.listdir(args.world)):
|
||||||
|
if name in ("region", "entities", "poi"):
|
||||||
|
continue
|
||||||
|
if args.minimal and name not in ("level.dat", "level.dat_old"):
|
||||||
|
continue
|
||||||
|
src, dst = os.path.join(args.world, name), os.path.join(args.out, name)
|
||||||
|
if os.path.exists(dst):
|
||||||
|
continue
|
||||||
|
if os.path.isdir(src):
|
||||||
|
shutil.copytree(src, dst, ignore=ignore)
|
||||||
|
else:
|
||||||
|
shutil.copy2(src, dst)
|
||||||
|
|
||||||
|
total_kept = total_dropped = 0
|
||||||
|
seen_regions = set()
|
||||||
|
failures: list[tuple[str, str]] = []
|
||||||
|
failed_regions: set[tuple[int, int]] = set()
|
||||||
|
for name in sorted(os.listdir(in_region)):
|
||||||
|
if not (name.startswith("r.") and name.endswith(".mca")):
|
||||||
|
continue
|
||||||
|
_, rx, rz, _ = name.split(".")
|
||||||
|
coords = (int(rx), int(rz))
|
||||||
|
keep = by_region.get(coords)
|
||||||
|
if keep is None:
|
||||||
|
continue # region untouched by the reveal set
|
||||||
|
try:
|
||||||
|
stats = prune_region(
|
||||||
|
os.path.join(in_region, name),
|
||||||
|
keep,
|
||||||
|
os.path.join(out_region, name),
|
||||||
|
args.dry_run,
|
||||||
|
)
|
||||||
|
except Exception as exc:
|
||||||
|
# A live server can rewrite a region file mid-read. A torn read is not a reason to
|
||||||
|
# abandon the whole pass - no output is written for a region that fails, so it can
|
||||||
|
# simply be retried.
|
||||||
|
failures.append((name, str(exc)))
|
||||||
|
failed_regions.add(coords)
|
||||||
|
print(f" ! {name}: {exc}", file=sys.stderr)
|
||||||
|
continue
|
||||||
|
seen_regions.add(coords)
|
||||||
|
total_kept += stats["kept"]
|
||||||
|
total_dropped += stats["dropped"]
|
||||||
|
if stats["external_payloads"]:
|
||||||
|
print(f" ! {name}: {stats['external_payloads']} external (.mcc) payload(s) "
|
||||||
|
f"referenced but not copied", file=sys.stderr)
|
||||||
|
print(f" {name}: kept {stats['kept']:>4} / dropped {stats['dropped']:>4}")
|
||||||
|
|
||||||
|
# Any region the player revealed but that has no .mca file.
|
||||||
|
missing = sorted(set(by_region) - seen_regions - failed_regions)
|
||||||
|
if missing:
|
||||||
|
print(f" ! {len(missing)} revealed region(s) have no .mca in the save "
|
||||||
|
f"(e.g. {missing[:3]})", file=sys.stderr)
|
||||||
|
|
||||||
|
verb = "would keep" if args.dry_run else "kept"
|
||||||
|
print(f"\n{verb} {total_kept:,} chunks, dropped {total_dropped:,}, "
|
||||||
|
f"across {len(seen_regions)} region file(s) -> {out_region}")
|
||||||
|
if failures:
|
||||||
|
print(f"\n{len(failures)} region(s) could not be read (not written):", file=sys.stderr)
|
||||||
|
for name, why in failures:
|
||||||
|
print(f" {name}: {why}", file=sys.stderr)
|
||||||
|
print("Re-run to retry them - usually a transient write by a running server.", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
Reference in new issue
Block a user