logan
5919955548
Move the completed 2026-09 candidate review (MODLIST-REVIEW/-GROUPS, CANDIDATE-APPENDIX, PACK-CONTEXT), FINDINGS.md and the shipped FTB Chunks swap write-up into analysis/archive/, with a README mapping their old paths. The review's build pieces, subagent intermediates and Modrinth API dumps are deleted: the assembled reports carry their content. MODLIST-GROUPS.md was a byte-for-byte duplicate of analysis/groups-table.md. Also: drop the now-pointless FINDINGS/MODLIST packwizignore lines (analysis/ covers the archive), and gitignore *.tar.gz so the 650 MB map-migration world tarball can't be committed by accident.
215 lines
12 KiB
Markdown
215 lines
12 KiB
Markdown
# FTB Chunks reveal -> Xaero's World Map: what transfers and how
|
|
|
|
**Status:** mask extraction implemented and validated against real data; world-save pruning implemented
|
|
and unit-tested; final step blocked on a copy of the server world save.
|
|
**Goal:** give each player a native Xaero's map covering, as closely as possible, exactly the terrain
|
|
they had revealed in FTB Chunks.
|
|
Confidence tags as in `analysis/archive/FINDINGS.md`.
|
|
|
|
---
|
|
|
|
## 1. Why we do not convert the pixels
|
|
|
|
FTB Chunks does **not** store a rendered map image. Verified by decompiling the pack's exact build,
|
|
`ftb-chunks-neoforge-2101.1.22.jar` (class strings in `MapManager`, `MapRegionData`, `MapDimension`,
|
|
`MapChunk`):
|
|
|
|
```
|
|
<game>/local/ftbchunks/data/<serverUUID>/
|
|
├── dimensions.txt, block_map.txt, biome_map.txt
|
|
└── <dimension>/
|
|
└── <XXXXX-XXXXX>.zip # region = 512x512 blocks; name = region coord + 60000, %05X-%05X
|
|
├── chunks.dat # the chunks this player has actually seen
|
|
├── data.png # NOT viewable: (height << 16) | (water|light|biome) per pixel
|
|
├── foliage.png # biome-tinted foliage RGB
|
|
├── grass.png # biome-tinted grass RGB
|
|
├── water.png # biome-tinted water RGB
|
|
└── blocks.png # FTB's internal 24-bit block colour index
|
|
```
|
|
|
|
The colour a player actually saw is computed at runtime by combining `blocks.png`'s index with FTB's own
|
|
`BlockColor` model via `block_map.txt`. Reproducing it means reimplementing FTB Chunks' renderer, and the
|
|
result would look like FTB's map - i.e. worse than native Xaero's. So we take the **reveal footprint**
|
|
from FTB Chunks and the **pixels** from Xaero's.
|
|
|
|
## 2. Why we also do not write Xaero region files
|
|
|
|
The current Xaero's region format is not the format the community tools understand. Comparing a real
|
|
`region.xaero` from Xaero's World Map 1.46.0 (extracted from the client instance) with the format
|
|
`JMtoXaero/RegionToXaero` writes:
|
|
|
|
| | old format (RegionToXaero) | real 1.46.0 |
|
|
| --- | --- | --- |
|
|
| header | `FF` then int 4 | `FF 00 06 00 08` + varying byte (format 6) |
|
|
| pixel payload | 4-byte parameters (+ optional state / colour / biome) | embeds NBT compounds |
|
|
| size for region `0,0` | ~25 KB | **355 KB** |
|
|
|
|
The 1.46.0 payload contains literal NBT - e.g. `0A 00 00 08 00 04 "Name" 00 10 "minecraft:gravel"` - so
|
|
each pixel now carries a block descriptor. This matches `JMtoXaero`'s own admission that its region
|
|
**merger** broke once Xaero changed the format. Writing this format is a live reverse-engineering project
|
|
against a closed, obfuscated, moving target. Not worth it.
|
|
|
|
**Therefore: drive Xaero's own generator, and control what it can see.**
|
|
|
|
## 3. The mechanism
|
|
|
|
Both mods use the same geometry - a region is 512x512 blocks = 32x32 chunks - so an FTB region maps 1:1
|
|
onto one Anvil region file, and an FTB within-region chunk `(x,z)` maps 1:1 onto Anvil slot
|
|
`x + z*32`. That makes the transplant exact rather than approximate:
|
|
|
|
1. Extract each player's reveal set from their `chunks.dat` files.
|
|
2. Prune a copy of the world save to exactly those chunks (Anvil header/sector surgery; chunk payloads
|
|
copied byte-for-byte, no NBT re-encoding).
|
|
3. The player opens the pruned save in **singleplayer** with the pack; Xaero's maps it directly from the
|
|
save, so absent chunks simply never appear on the map.
|
|
4. Copy the generated instance
|
|
(`xaero/world-map/World/null/<map>/`) into the server's folder
|
|
(`xaero/world-map/Multiplayer_advilplug.top/<DIM>/`).
|
|
|
|
Result: **per-chunk-exact reveal extent**, rendered natively by Xaero's. Residual differences are only
|
|
the colour palette/shading (Xaero's, not FTB's) - which is the whole point.
|
|
|
|
## 4. What the real data says
|
|
|
|
Measured on the supplied archive - **Stirwick**, `minecraft:overworld`:
|
|
|
|
| Metric | Value |
|
|
| --- | --- |
|
|
| Region files | 167 |
|
|
| Revealed chunks | **98,170** (of 171,008 in the bounding box) |
|
|
| Region extent | x `-9..9`, z `-14..7` |
|
|
| Block extent | x `-4608..5120`, z `-7168..4096` |
|
|
| Fully revealed regions (`1024/1024`) | 10 |
|
|
| Median region fill | 560 / 1024 (~55%) |
|
|
| Sparse regions (`<64` chunks) | 10 |
|
|
| `chunks.dat` vs `data.png` disagreements | **0** |
|
|
|
|
Two consequences:
|
|
|
|
- **`chunks.dat` alone is a trustworthy mask.** It was cross-checked against the `data.png` height
|
|
sentinel (`HeightUtils.UNKNOWN = Short.MIN_VALUE + 1 = -32767`) for all 167 regions. Zero
|
|
disagreements. Suspicion that claimed-but-never-visited chunks would inflate the set did not
|
|
materialise.
|
|
- **Region-granularity pruning is not good enough.** The median region is only ~55% explored, and 10
|
|
regions are under 6%. Copying whole `.mca` files a player touched would reveal roughly twice the
|
|
terrain they had seen. Chunk-level pruning is worth the extra ~40 lines.
|
|
|
|
`data.png` decoding detail, for anyone revisiting this: FTB writes `(height << 16) | wlb` into a
|
|
`TYPE_INT_ARGB` image, so after PNG round-trip the 16-bit height is `(A << 8) | R` and the
|
|
water/light/biome word is `(G << 8) | B`. Unrevealed pixels are exactly `A=0x80, R=0x01`.
|
|
|
|
## 5. Tooling
|
|
|
|
Both scripts are stdlib-only and live in `tools/` (which `.packwizignore` excludes, so they never ship).
|
|
|
|
`tools/migrate-player.sh` - the driver, does steps 1-2 and prints the manual remainder:
|
|
|
|
```
|
|
tools/migrate-player.sh Stirwick <ftb-data-or-archive> <world-dir> \
|
|
--dimension minecraft:overworld --work /var/tmp/xaero-migration
|
|
```
|
|
|
|
`tools/ftbchunks-reveal.py` - **validated against Stirwick's real 167-region archive.**
|
|
Emits a JSON mask of global chunk coordinates per dimension. Pass `--dimension` when the input is a
|
|
bare directory/zip of region files, which cannot name its own dimension.
|
|
|
|
```
|
|
tools/ftbchunks-reveal.py <ftb-data-or-archive> --out mask.json --dimension minecraft:overworld --verify
|
|
```
|
|
|
|
`tools/filter-xaero-regions.py` - post-harvest cleanup. Compares the Xaero's map folder against the
|
|
mask and removes regions the player never revealed (i.e. terrain Minecraft regenerated around the
|
|
login position). File-name-only, so no Xaero format knowledge needed; reports before deleting.
|
|
|
|
```
|
|
tools/filter-xaero-regions.py <game>/xaero/world-map/Multiplayer_advilplug.top/null/ mask.json
|
|
```
|
|
|
|
`tools/prune-world-to-reveal.py` - **validated end-to-end against a real region file from the server**
|
|
(`world/region/r.1.-2.mca`, 15.7 MB):
|
|
|
|
| Check | Result |
|
|
| --- | --- |
|
|
| Anvil parse (1024 slots, all zlib, 2.5-21.8 KB payloads) | OK |
|
|
| Slot -> chunk mapping (`slot = x + z*32`), confirmed by reading `xPos`/`zPos` NBT | **1024 / 1024 exact** |
|
|
| Prune against that player's FTB mask (52 of 1024 chunks revealed) | 52 kept / 972 dropped |
|
|
| Output sector-aligned | yes |
|
|
| Kept slot set == mask | yes |
|
|
| Kept payloads + timestamps byte-identical | yes |
|
|
| Kept chunks re-parse with correct coordinates | 52 / 52 |
|
|
| Size | 15,745,024 -> 471,040 bytes (3.0%) |
|
|
|
|
Earlier synthetic tests also cover nested per-dimension `region`/`entities`/`poi` trees not being
|
|
duplicated.
|
|
|
|
```
|
|
tools/prune-world-to-reveal.py <world-copy> mask.json <out-dir> --dimension minecraft:overworld
|
|
```
|
|
|
|
`--minimal` writes only `level.dat` plus the pruned `region/` tree - the smallest thing Xaero's can
|
|
open and map, and the best fit when the full save is too large to copy around. `--dry-run` reports
|
|
kept/dropped counts per region without writing anything. Runtime is ~0.15 s per 15 MB region.
|
|
|
|
## 6. Running it where the save lives
|
|
|
|
The tools are stdlib-only Python, so nothing has to be transferred: run them on the machine that holds
|
|
the world. The live world is **not** on the VPS (the VPS is only a proxy in front of the real server),
|
|
so this has to run where the save actually lives. The per-player footprint is only the pruned `region/`
|
|
tree for the regions that player revealed, not the whole save.
|
|
|
|
```bash
|
|
tools/migrate-player.sh Stirwick <stirwick-ftbchunks-data> <world-dir> \
|
|
--dimension minecraft:overworld --work /var/tmp/xaero-migration
|
|
```
|
|
|
|
That extracts the mask (with the `chunks.dat` vs `data.png` cross-check) and prunes the world,
|
|
dry-run first. Then, by hand:
|
|
|
|
3. Copy the pruned world (`<work>/<player>/sp-world`) to a machine that can run the pack and open it in
|
|
**singleplayer** once. Let Xaero's map it, then quit.
|
|
4. Move the generated instance, `xaero/world-map/World/null/<map>/`, into the server's map folder,
|
|
`xaero/world-map/Multiplayer_advilplug.top/<DIM>/`.
|
|
5. Run `tools/filter-xaero-regions.py` on that folder to drop regions the player never revealed.
|
|
6. Delete the scratch directory - usually a few hundred MB.
|
|
|
|
### Two things that will bite
|
|
|
|
- **The server does not need stopping, but freeze saves while it runs.** The pruner only reads the world
|
|
and writes to a separate output directory. However a running server rewrites region files on autosave
|
|
and rewrites `level.dat` in place, so a read can tear - and half a `level.dat` is an unopenable
|
|
singleplayer world. From the console: `save-off` -> dry run -> real run -> `save-all` -> `save-on`.
|
|
The pass takes a couple of minutes. Never forget `save-on`, and never point `--out` at the live world.
|
|
A region that does fail to read is reported, skipped, and left unwritten; re-run to retry it.
|
|
|
|
- **Opening the world regenerates chunks near the login position.** Minecraft loads and generates the
|
|
chunks around wherever the player spawns, so terrain the player never revealed can reappear on the
|
|
map if it happens to sit near world spawn. It is bounded by render distance (well under one
|
|
512x512 region at typical settings), and it is easy to undo: step 4's cleanup compares the harvested
|
|
region file names against the mask and deletes any region the player never revealed. Keeping
|
|
`level.dat`'s spawn inside the revealed area, or not walking around in step 3, avoids the rest.
|
|
- **Do not copy the world with hardlinks.** `cp -al` would make the singleplayer client's own writes to
|
|
`level.dat` land in the live server save. A real copy, or `--minimal` into a fresh directory, is safe.
|
|
|
|
## 7. Open items
|
|
|
|
- **Needs: the full world save, read in place on the server.** One real region file (`r.1.-2.mca`) has
|
|
now validated the reader, the slot mapping and the writer, so the remaining unknowns are operational
|
|
rather than format-related: total runtime over 167 regions (extrapolating ~0.15 s per 15 MB region,
|
|
a couple of minutes) and the size of the pruned output. A full 1024-chunk region on this server is
|
|
~15 MB; the player revealed 57.4% of the bounding box overall, so expect the pruned per-player world
|
|
to land in the low hundreds of MB. It is scratch data and can be deleted once the Xaero's map has
|
|
been harvested.
|
|
- **Per-player, per-dimension.** Only `minecraft:overworld` was supplied and only one player's data.
|
|
Modded dimensions will each need an FTB-data -> Xaero-dimension mapping. Xaero names its dimension
|
|
folders `null`/`DIM-1`/`DIM1` for vanilla and uses the `minecraft$overworld` form (`$`, not `:`) in
|
|
`dimension_config.txt`; a `dimensions.txt` with more than one entry will confirm the rest.
|
|
- **Step 3 costs one client launch per player.** It cannot be done server-side; Xaero's maps from a save
|
|
on the client. ~10 launches for this pack.
|
|
- **Waypoints are a separate, easier win.** FTB stores them in `waypoints.json` next to the region data
|
|
(empty, 16 bytes, for the sampled player). Xaero waypoints are plain files under `xaero/minimap/`,
|
|
so a converter is straightforward - worth doing at the same time if anyone has waypoints to keep.
|
|
- **Fallbacks if the save never materialises:** region-granularity pruning (coarse, ~2x over-reveal), or
|
|
shipping the full "Convert All Regions" map to everyone (over-reveals everything, zero code).
|
|
- Nothing here is time-sensitive: removing the mod from the pack does not touch `local/ftbchunks/`, so
|
|
every client's reveal data is still on disk and this remains doable later.
|