onion-mc/analysis/FTB-XAERO-MAP-MIGRATION.md

# FTB Chunks reveal -> Xaero's World Map: what transfers and how

**Status:** mask extraction implemented and validated against real data; world-save pruning implemented
and unit-tested; final step blocked on a copy of the server world save.
**Goal:** give each player a native Xaero's map covering, as closely as possible, exactly the terrain
they had revealed in FTB Chunks.
Confidence tags as in `FINDINGS.md`.

---

## 1. Why we do not convert the pixels

FTB Chunks does **not** store a rendered map image. Verified by decompiling the pack's exact build,
`ftb-chunks-neoforge-2101.1.22.jar` (class strings in `MapManager`, `MapRegionData`, `MapDimension`,
`MapChunk`):

```
<game>/local/ftbchunks/data/<serverUUID>/
├── dimensions.txt, block_map.txt, biome_map.txt
└── <dimension>/
    └── <XXXXX-XXXXX>.zip          # region = 512x512 blocks; name = region coord + 60000, %05X-%05X
        ├── chunks.dat             # the chunks this player has actually seen
        ├── data.png               # NOT viewable: (height << 16) | (water|light|biome) per pixel
        ├── foliage.png            # biome-tinted foliage RGB
        ├── grass.png              # biome-tinted grass RGB
        ├── water.png              # biome-tinted water RGB
        └── blocks.png             # FTB's internal 24-bit block colour index
```

The colour a player actually saw is computed at runtime by combining `blocks.png`'s index with FTB's own
`BlockColor` model via `block_map.txt`. Reproducing it means reimplementing FTB Chunks' renderer, and the
result would look like FTB's map - i.e. worse than native Xaero's. So we take the **reveal footprint**
from FTB Chunks and the **pixels** from Xaero's.

## 2. Why we also do not write Xaero region files

The current Xaero's region format is not the format the community tools understand. Comparing a real
`region.xaero` from Xaero's World Map 1.46.0 (extracted from the client instance) with the format
`JMtoXaero/RegionToXaero` writes:

| | old format (RegionToXaero) | real 1.46.0 |
| --- | --- | --- |
| header | `FF` then int 4 | `FF 00 06 00 08` + varying byte (format 6) |
| pixel payload | 4-byte parameters (+ optional state / colour / biome) | embeds NBT compounds |
| size for region `0,0` | ~25 KB | **355 KB** |

The 1.46.0 payload contains literal NBT - e.g. `0A 00 00 08 00 04 "Name" 00 10 "minecraft:gravel"` - so
each pixel now carries a block descriptor. This matches `JMtoXaero`'s own admission that its region
**merger** broke once Xaero changed the format. Writing this format is a live reverse-engineering project
against a closed, obfuscated, moving target. Not worth it.

**Therefore: drive Xaero's own generator, and control what it can see.**

## 3. The mechanism

Both mods use the same geometry - a region is 512x512 blocks = 32x32 chunks - so an FTB region maps 1:1
onto one Anvil region file, and an FTB within-region chunk `(x,z)` maps 1:1 onto Anvil slot
`x + z*32`. That makes the transplant exact rather than approximate:

1. Extract each player's reveal set from their `chunks.dat` files.
2. Prune a copy of the world save to exactly those chunks (Anvil header/sector surgery; chunk payloads
   copied byte-for-byte, no NBT re-encoding).
3. The player opens the pruned save in **singleplayer** with the pack; Xaero's maps it directly from the
   save, so absent chunks simply never appear on the map.
4. Copy the generated instance
   (`xaero/world-map/World/null/<map>/`) into the server's folder
   (`xaero/world-map/Multiplayer_advilplug.top/<DIM>/`).

Result: **per-chunk-exact reveal extent**, rendered natively by Xaero's. Residual differences are only
the colour palette/shading (Xaero's, not FTB's) - which is the whole point.

## 4. What the real data says

Measured on the supplied archive - **Stirwick**, `minecraft:overworld`:

| Metric | Value |
| --- | --- |
| Region files | 167 |
| Revealed chunks | **98,170** (of 171,008 in the bounding box) |
| Region extent | x `-9..9`, z `-14..7` |
| Block extent | x `-4608..5120`, z `-7168..4096` |
| Fully revealed regions (`1024/1024`) | 10 |
| Median region fill | 560 / 1024 (~55%) |
| Sparse regions (`<64` chunks) | 10 |
| `chunks.dat` vs `data.png` disagreements | **0** |

Two consequences:

- **`chunks.dat` alone is a trustworthy mask.** It was cross-checked against the `data.png` height
  sentinel (`HeightUtils.UNKNOWN = Short.MIN_VALUE + 1 = -32767`) for all 167 regions. Zero
  disagreements. Suspicion that claimed-but-never-visited chunks would inflate the set did not
  materialise.
- **Region-granularity pruning is not good enough.** The median region is only ~55% explored, and 10
  regions are under 6%. Copying whole `.mca` files a player touched would reveal roughly twice the
  terrain they had seen. Chunk-level pruning is worth the extra ~40 lines.

`data.png` decoding detail, for anyone revisiting this: FTB writes `(height << 16) | wlb` into a
`TYPE_INT_ARGB` image, so after PNG round-trip the 16-bit height is `(A << 8) | R` and the
water/light/biome word is `(G << 8) | B`. Unrevealed pixels are exactly `A=0x80, R=0x01`.

## 5. Tooling

Both scripts are stdlib-only and live in `tools/` (which `.packwizignore` excludes, so they never ship).

`tools/migrate-player.sh` - the driver, does steps 1-2 and prints the manual remainder:

```
tools/migrate-player.sh Stirwick <ftb-data-or-archive> <world-dir> \
    --dimension minecraft:overworld --work /var/tmp/xaero-migration
```

`tools/ftbchunks-reveal.py` - **validated against Stirwick's real 167-region archive.**
Emits a JSON mask of global chunk coordinates per dimension. Pass `--dimension` when the input is a
bare directory/zip of region files, which cannot name its own dimension.

```
tools/ftbchunks-reveal.py <ftb-data-or-archive> --out mask.json --dimension minecraft:overworld --verify
```

`tools/filter-xaero-regions.py` - post-harvest cleanup. Compares the Xaero's map folder against the
mask and removes regions the player never revealed (i.e. terrain Minecraft regenerated around the
login position). File-name-only, so no Xaero format knowledge needed; reports before deleting.

```
tools/filter-xaero-regions.py <game>/xaero/world-map/Multiplayer_advilplug.top/null/ mask.json
```

`tools/prune-world-to-reveal.py` - **validated end-to-end against a real region file from the server**
(`world/region/r.1.-2.mca`, 15.7 MB):

| Check | Result |
| --- | --- |
| Anvil parse (1024 slots, all zlib, 2.5-21.8 KB payloads) | OK |
| Slot -> chunk mapping (`slot = x + z*32`), confirmed by reading `xPos`/`zPos` NBT | **1024 / 1024 exact** |
| Prune against that player's FTB mask (52 of 1024 chunks revealed) | 52 kept / 972 dropped |
| Output sector-aligned | yes |
| Kept slot set == mask | yes |
| Kept payloads + timestamps byte-identical | yes |
| Kept chunks re-parse with correct coordinates | 52 / 52 |
| Size | 15,745,024 -> 471,040 bytes (3.0%) |

Earlier synthetic tests also cover nested per-dimension `region`/`entities`/`poi` trees not being
duplicated.

```
tools/prune-world-to-reveal.py <world-copy> mask.json <out-dir> --dimension minecraft:overworld
```

`--minimal` writes only `level.dat` plus the pruned `region/` tree - the smallest thing Xaero's can
open and map, and the best fit when the full save is too large to copy around. `--dry-run` reports
kept/dropped counts per region without writing anything. Runtime is ~0.15 s per 15 MB region.

## 6. Running it where the save lives

The tools are stdlib-only Python, so nothing has to be transferred: run them on the machine that holds
the world. The live world is **not** on the VPS (the VPS is only a proxy in front of the real server),
so this has to run where the save actually lives. The per-player footprint is only the pruned `region/`
tree for the regions that player revealed, not the whole save.

```bash
tools/migrate-player.sh Stirwick <stirwick-ftbchunks-data> <world-dir> \
    --dimension minecraft:overworld --work /var/tmp/xaero-migration
```

That extracts the mask (with the `chunks.dat` vs `data.png` cross-check) and prunes the world,
dry-run first. Then, by hand:

3. Copy the pruned world (`<work>/<player>/sp-world`) to a machine that can run the pack and open it in
   **singleplayer** once. Let Xaero's map it, then quit.
4. Move the generated instance, `xaero/world-map/World/null/<map>/`, into the server's map folder,
   `xaero/world-map/Multiplayer_advilplug.top/<DIM>/`.
5. Run `tools/filter-xaero-regions.py` on that folder to drop regions the player never revealed.
6. Delete the scratch directory - usually a few hundred MB.

### Two things that will bite

- **The server does not need stopping, but freeze saves while it runs.** The pruner only reads the world
  and writes to a separate output directory. However a running server rewrites region files on autosave
  and rewrites `level.dat` in place, so a read can tear - and half a `level.dat` is an unopenable
  singleplayer world. From the console: `save-off` -> dry run -> real run -> `save-all` -> `save-on`.
  The pass takes a couple of minutes. Never forget `save-on`, and never point `--out` at the live world.
  A region that does fail to read is reported, skipped, and left unwritten; re-run to retry it.

- **Opening the world regenerates chunks near the login position.** Minecraft loads and generates the
  chunks around wherever the player spawns, so terrain the player never revealed can reappear on the
  map if it happens to sit near world spawn. It is bounded by render distance (well under one
  512x512 region at typical settings), and it is easy to undo: step 4's cleanup compares the harvested
  region file names against the mask and deletes any region the player never revealed. Keeping
  `level.dat`'s spawn inside the revealed area, or not walking around in step 3, avoids the rest.
- **Do not copy the world with hardlinks.** `cp -al` would make the singleplayer client's own writes to
  `level.dat` land in the live server save. A real copy, or `--minimal` into a fresh directory, is safe.

## 7. Open items

- **Needs: the full world save, read in place on the server.** One real region file (`r.1.-2.mca`) has
  now validated the reader, the slot mapping and the writer, so the remaining unknowns are operational
  rather than format-related: total runtime over 167 regions (extrapolating ~0.15 s per 15 MB region,
  a couple of minutes) and the size of the pruned output. A full 1024-chunk region on this server is
  ~15 MB; the player revealed 57.4% of the bounding box overall, so expect the pruned per-player world
  to land in the low hundreds of MB. It is scratch data and can be deleted once the Xaero's map has
  been harvested.
- **Per-player, per-dimension.** Only `minecraft:overworld` was supplied and only one player's data.
  Modded dimensions will each need an FTB-data -> Xaero-dimension mapping. Xaero names its dimension
  folders `null`/`DIM-1`/`DIM1` for vanilla and uses the `minecraft$overworld` form (`$`, not `:`) in
  `dimension_config.txt`; a `dimensions.txt` with more than one entry will confirm the rest.
- **Step 3 costs one client launch per player.** It cannot be done server-side; Xaero's maps from a save
  on the client. ~10 launches for this pack.
- **Waypoints are a separate, easier win.** FTB stores them in `waypoints.json` next to the region data
  (empty, 16 bytes, for the sampled player). Xaero waypoints are plain files under `xaero/minimap/`,
  so a converter is straightforward - worth doing at the same time if anyone has waypoints to keep.
- **Fallbacks if the save never materialises:** region-granularity pruning (coarse, ~2x over-reveal), or
  shipping the full "Convert All Regions" map to everyone (over-reveals everything, zero code).
- Nothing here is time-sensitive: removing the mod from the pack does not touch `local/ftbchunks/`, so
  every client's reveal data is still on disk and this remains doable later.