logan
769af6fa2d
The swap does not carry players' explored terrain across. FTB Chunks stores no rendered map (its region zips hold chunks.dat plus five data PNGs, one of which packs height into an image channel), and Xaero's region format is closed and now version 6 with per-pixel NBT blobs, so the community converters no longer work. Rather than convert pixels, prune a copy of the world save down to exactly the chunks a player revealed and let Xaero's own generator map it. Validated against real data: extracting the 167-region / 98,170-chunk mask cross-checks clean against FTB's own pixel data, and Anvil pruning round-trips byte-identically (15.7 MB -> 471 KB on a live region file, slot-to-chunk mapping confirmed against 1024 chunk NBT headers). Deferred for now: needs the world save and one singleplayer pass per player. See analysis/FTB-XAERO-MAP-MIGRATION.md.
215 lines
12 KiB
Markdown
215 lines
12 KiB
Markdown
# FTB Chunks reveal -> Xaero's World Map: what transfers and how
|
|
|
|
**Status:** mask extraction implemented and validated against real data; world-save pruning implemented
|
|
and unit-tested; final step blocked on a copy of the server world save.
|
|
**Goal:** give each player a native Xaero's map covering, as closely as possible, exactly the terrain
|
|
they had revealed in FTB Chunks.
|
|
Confidence tags as in `FINDINGS.md`.
|
|
|
|
---
|
|
|
|
## 1. Why we do not convert the pixels
|
|
|
|
FTB Chunks does **not** store a rendered map image. Verified by decompiling the pack's exact build,
|
|
`ftb-chunks-neoforge-2101.1.22.jar` (class strings in `MapManager`, `MapRegionData`, `MapDimension`,
|
|
`MapChunk`):
|
|
|
|
```
|
|
<game>/local/ftbchunks/data/<serverUUID>/
|
|
├── dimensions.txt, block_map.txt, biome_map.txt
|
|
└── <dimension>/
|
|
└── <XXXXX-XXXXX>.zip # region = 512x512 blocks; name = region coord + 60000, %05X-%05X
|
|
├── chunks.dat # the chunks this player has actually seen
|
|
├── data.png # NOT viewable: (height << 16) | (water|light|biome) per pixel
|
|
├── foliage.png # biome-tinted foliage RGB
|
|
├── grass.png # biome-tinted grass RGB
|
|
├── water.png # biome-tinted water RGB
|
|
└── blocks.png # FTB's internal 24-bit block colour index
|
|
```
|
|
|
|
The colour a player actually saw is computed at runtime by combining `blocks.png`'s index with FTB's own
|
|
`BlockColor` model via `block_map.txt`. Reproducing it means reimplementing FTB Chunks' renderer, and the
|
|
result would look like FTB's map - i.e. worse than native Xaero's. So we take the **reveal footprint**
|
|
from FTB Chunks and the **pixels** from Xaero's.
|
|
|
|
## 2. Why we also do not write Xaero region files
|
|
|
|
The current Xaero's region format is not the format the community tools understand. Comparing a real
|
|
`region.xaero` from Xaero's World Map 1.46.0 (extracted from the client instance) with the format
|
|
`JMtoXaero/RegionToXaero` writes:
|
|
|
|
| | old format (RegionToXaero) | real 1.46.0 |
|
|
| --- | --- | --- |
|
|
| header | `FF` then int 4 | `FF 00 06 00 08` + varying byte (format 6) |
|
|
| pixel payload | 4-byte parameters (+ optional state / colour / biome) | embeds NBT compounds |
|
|
| size for region `0,0` | ~25 KB | **355 KB** |
|
|
|
|
The 1.46.0 payload contains literal NBT - e.g. `0A 00 00 08 00 04 "Name" 00 10 "minecraft:gravel"` - so
|
|
each pixel now carries a block descriptor. This matches `JMtoXaero`'s own admission that its region
|
|
**merger** broke once Xaero changed the format. Writing this format is a live reverse-engineering project
|
|
against a closed, obfuscated, moving target. Not worth it.
|
|
|
|
**Therefore: drive Xaero's own generator, and control what it can see.**
|
|
|
|
## 3. The mechanism
|
|
|
|
Both mods use the same geometry - a region is 512x512 blocks = 32x32 chunks - so an FTB region maps 1:1
|
|
onto one Anvil region file, and an FTB within-region chunk `(x,z)` maps 1:1 onto Anvil slot
|
|
`x + z*32`. That makes the transplant exact rather than approximate:
|
|
|
|
1. Extract each player's reveal set from their `chunks.dat` files.
|
|
2. Prune a copy of the world save to exactly those chunks (Anvil header/sector surgery; chunk payloads
|
|
copied byte-for-byte, no NBT re-encoding).
|
|
3. The player opens the pruned save in **singleplayer** with the pack; Xaero's maps it directly from the
|
|
save, so absent chunks simply never appear on the map.
|
|
4. Copy the generated instance
|
|
(`xaero/world-map/World/null/<map>/`) into the server's folder
|
|
(`xaero/world-map/Multiplayer_advilplug.top/<DIM>/`).
|
|
|
|
Result: **per-chunk-exact reveal extent**, rendered natively by Xaero's. Residual differences are only
|
|
the colour palette/shading (Xaero's, not FTB's) - which is the whole point.
|
|
|
|
## 4. What the real data says
|
|
|
|
Measured on the supplied archive - **Stirwick**, `minecraft:overworld`:
|
|
|
|
| Metric | Value |
|
|
| --- | --- |
|
|
| Region files | 167 |
|
|
| Revealed chunks | **98,170** (of 171,008 in the bounding box) |
|
|
| Region extent | x `-9..9`, z `-14..7` |
|
|
| Block extent | x `-4608..5120`, z `-7168..4096` |
|
|
| Fully revealed regions (`1024/1024`) | 10 |
|
|
| Median region fill | 560 / 1024 (~55%) |
|
|
| Sparse regions (`<64` chunks) | 10 |
|
|
| `chunks.dat` vs `data.png` disagreements | **0** |
|
|
|
|
Two consequences:
|
|
|
|
- **`chunks.dat` alone is a trustworthy mask.** It was cross-checked against the `data.png` height
|
|
sentinel (`HeightUtils.UNKNOWN = Short.MIN_VALUE + 1 = -32767`) for all 167 regions. Zero
|
|
disagreements. Suspicion that claimed-but-never-visited chunks would inflate the set did not
|
|
materialise.
|
|
- **Region-granularity pruning is not good enough.** The median region is only ~55% explored, and 10
|
|
regions are under 6%. Copying whole `.mca` files a player touched would reveal roughly twice the
|
|
terrain they had seen. Chunk-level pruning is worth the extra ~40 lines.
|
|
|
|
`data.png` decoding detail, for anyone revisiting this: FTB writes `(height << 16) | wlb` into a
|
|
`TYPE_INT_ARGB` image, so after PNG round-trip the 16-bit height is `(A << 8) | R` and the
|
|
water/light/biome word is `(G << 8) | B`. Unrevealed pixels are exactly `A=0x80, R=0x01`.
|
|
|
|
## 5. Tooling
|
|
|
|
Both scripts are stdlib-only and live in `tools/` (which `.packwizignore` excludes, so they never ship).
|
|
|
|
`tools/migrate-player.sh` - the driver, does steps 1-2 and prints the manual remainder:
|
|
|
|
```
|
|
tools/migrate-player.sh Stirwick <ftb-data-or-archive> <world-dir> \
|
|
--dimension minecraft:overworld --work /var/tmp/xaero-migration
|
|
```
|
|
|
|
`tools/ftbchunks-reveal.py` - **validated against Stirwick's real 167-region archive.**
|
|
Emits a JSON mask of global chunk coordinates per dimension. Pass `--dimension` when the input is a
|
|
bare directory/zip of region files, which cannot name its own dimension.
|
|
|
|
```
|
|
tools/ftbchunks-reveal.py <ftb-data-or-archive> --out mask.json --dimension minecraft:overworld --verify
|
|
```
|
|
|
|
`tools/filter-xaero-regions.py` - post-harvest cleanup. Compares the Xaero's map folder against the
|
|
mask and removes regions the player never revealed (i.e. terrain Minecraft regenerated around the
|
|
login position). File-name-only, so no Xaero format knowledge needed; reports before deleting.
|
|
|
|
```
|
|
tools/filter-xaero-regions.py <game>/xaero/world-map/Multiplayer_advilplug.top/null/ mask.json
|
|
```
|
|
|
|
`tools/prune-world-to-reveal.py` - **validated end-to-end against a real region file from the server**
|
|
(`world/region/r.1.-2.mca`, 15.7 MB):
|
|
|
|
| Check | Result |
|
|
| --- | --- |
|
|
| Anvil parse (1024 slots, all zlib, 2.5-21.8 KB payloads) | OK |
|
|
| Slot -> chunk mapping (`slot = x + z*32`), confirmed by reading `xPos`/`zPos` NBT | **1024 / 1024 exact** |
|
|
| Prune against that player's FTB mask (52 of 1024 chunks revealed) | 52 kept / 972 dropped |
|
|
| Output sector-aligned | yes |
|
|
| Kept slot set == mask | yes |
|
|
| Kept payloads + timestamps byte-identical | yes |
|
|
| Kept chunks re-parse with correct coordinates | 52 / 52 |
|
|
| Size | 15,745,024 -> 471,040 bytes (3.0%) |
|
|
|
|
Earlier synthetic tests also cover nested per-dimension `region`/`entities`/`poi` trees not being
|
|
duplicated.
|
|
|
|
```
|
|
tools/prune-world-to-reveal.py <world-copy> mask.json <out-dir> --dimension minecraft:overworld
|
|
```
|
|
|
|
`--minimal` writes only `level.dat` plus the pruned `region/` tree - the smallest thing Xaero's can
|
|
open and map, and the best fit when the full save is too large to copy around. `--dry-run` reports
|
|
kept/dropped counts per region without writing anything. Runtime is ~0.15 s per 15 MB region.
|
|
|
|
## 6. Running it where the save lives
|
|
|
|
The tools are stdlib-only Python, so nothing has to be transferred: run them on the machine that holds
|
|
the world. The live world is **not** on the VPS (the VPS is only a proxy in front of the real server),
|
|
so this has to run where the save actually lives. The per-player footprint is only the pruned `region/`
|
|
tree for the regions that player revealed, not the whole save.
|
|
|
|
```bash
|
|
tools/migrate-player.sh Stirwick <stirwick-ftbchunks-data> <world-dir> \
|
|
--dimension minecraft:overworld --work /var/tmp/xaero-migration
|
|
```
|
|
|
|
That extracts the mask (with the `chunks.dat` vs `data.png` cross-check) and prunes the world,
|
|
dry-run first. Then, by hand:
|
|
|
|
3. Copy the pruned world (`<work>/<player>/sp-world`) to a machine that can run the pack and open it in
|
|
**singleplayer** once. Let Xaero's map it, then quit.
|
|
4. Move the generated instance, `xaero/world-map/World/null/<map>/`, into the server's map folder,
|
|
`xaero/world-map/Multiplayer_advilplug.top/<DIM>/`.
|
|
5. Run `tools/filter-xaero-regions.py` on that folder to drop regions the player never revealed.
|
|
6. Delete the scratch directory - usually a few hundred MB.
|
|
|
|
### Two things that will bite
|
|
|
|
- **The server does not need stopping, but freeze saves while it runs.** The pruner only reads the world
|
|
and writes to a separate output directory. However a running server rewrites region files on autosave
|
|
and rewrites `level.dat` in place, so a read can tear - and half a `level.dat` is an unopenable
|
|
singleplayer world. From the console: `save-off` -> dry run -> real run -> `save-all` -> `save-on`.
|
|
The pass takes a couple of minutes. Never forget `save-on`, and never point `--out` at the live world.
|
|
A region that does fail to read is reported, skipped, and left unwritten; re-run to retry it.
|
|
|
|
- **Opening the world regenerates chunks near the login position.** Minecraft loads and generates the
|
|
chunks around wherever the player spawns, so terrain the player never revealed can reappear on the
|
|
map if it happens to sit near world spawn. It is bounded by render distance (well under one
|
|
512x512 region at typical settings), and it is easy to undo: step 4's cleanup compares the harvested
|
|
region file names against the mask and deletes any region the player never revealed. Keeping
|
|
`level.dat`'s spawn inside the revealed area, or not walking around in step 3, avoids the rest.
|
|
- **Do not copy the world with hardlinks.** `cp -al` would make the singleplayer client's own writes to
|
|
`level.dat` land in the live server save. A real copy, or `--minimal` into a fresh directory, is safe.
|
|
|
|
## 7. Open items
|
|
|
|
- **Needs: the full world save, read in place on the server.** One real region file (`r.1.-2.mca`) has
|
|
now validated the reader, the slot mapping and the writer, so the remaining unknowns are operational
|
|
rather than format-related: total runtime over 167 regions (extrapolating ~0.15 s per 15 MB region,
|
|
a couple of minutes) and the size of the pruned output. A full 1024-chunk region on this server is
|
|
~15 MB; the player revealed 57.4% of the bounding box overall, so expect the pruned per-player world
|
|
to land in the low hundreds of MB. It is scratch data and can be deleted once the Xaero's map has
|
|
been harvested.
|
|
- **Per-player, per-dimension.** Only `minecraft:overworld` was supplied and only one player's data.
|
|
Modded dimensions will each need an FTB-data -> Xaero-dimension mapping. Xaero names its dimension
|
|
folders `null`/`DIM-1`/`DIM1` for vanilla and uses the `minecraft$overworld` form (`$`, not `:`) in
|
|
`dimension_config.txt`; a `dimensions.txt` with more than one entry will confirm the rest.
|
|
- **Step 3 costs one client launch per player.** It cannot be done server-side; Xaero's maps from a save
|
|
on the client. ~10 launches for this pack.
|
|
- **Waypoints are a separate, easier win.** FTB stores them in `waypoints.json` next to the region data
|
|
(empty, 16 bytes, for the sampled player). Xaero waypoints are plain files under `xaero/minimap/`,
|
|
so a converter is straightforward - worth doing at the same time if anyone has waypoints to keep.
|
|
- **Fallbacks if the save never materialises:** region-granularity pruning (coarse, ~2x over-reveal), or
|
|
shipping the full "Convert All Regions" map to everyone (over-reveals everything, zero code).
|
|
- Nothing here is time-sensitive: removing the mod from the pack does not touch `local/ftbchunks/`, so
|
|
every client's reveal data is still on disk and this remains doable later.
|