cosmonarchy-bw-prerelease/tools/tbl/README.md

# tbl

A round-trip compiler between StarCraft: Brood War `.tbl` string tables and a
plain-text `.tbl.txt` source format. TypeScript, [Effect](https://effect.website) v4, Bun.

Every `.tbl` shipped in this repo decompiles and recompiles **byte for byte
identically** — that is enforced by the test suite.

## Usage

```sh
bun install

bun run tbl decompile ../../mpq/arr/images.tbl   # -> images.tbl.txt
$EDITOR ../../mpq/arr/images.tbl.txt
bun run tbl compile   ../../mpq/arr/images.tbl.txt   # -> images.tbl

bun run tbl verify ../../mpq/rez/stat_txt.tbl    # prove a round trip is lossless
bun run tbl info   ../../mpq/rez/stat_txt.tbl    # entry counts, offset headroom
```

| Command | Flags |
| --- | --- |
| `decompile <in.tbl>` | `-o <out>`, `--annotate` (`# [index]` above each entry), `--bare` (no banner), `--dialect` |
| `compile <in.tbl.txt>` | `-o <out>`, `--dedupe` (share identical entries), `--dialect` |
| `build [root]` | `--check` (fail instead of writing), `--dedupe` |
| `verify <in.tbl>` | — |
| `info <in.tbl>` | — |

`build` walks a directory for `.tbl.txt` sources and recompiles each one in place —
the whole `mpq` tree in one go:

```sh
bun run build      # recompile every table under ../../mpq
bun run check      # fail if any committed .tbl is out of date
```

`tools/hooks/pre-commit` runs `check` so a stale `.tbl` cannot be committed beside an
edited `.tbl.txt`. Install it with
`ln -sf ../../tools/hooks/pre-commit .git/hooks/pre-commit`.

`decompile` defaults to `<input>.txt`, so `images.tbl` becomes `images.tbl.txt`
and never lands on the unrelated `images.txt` that DatEdit uses for its name
lists. `compile` strips that `.txt` back off.

## The binary format

```
u16le  count
u16le  offset[count]     // absolute, from the start of the file
...    NUL-terminated string data
```

One index does not always mean one string. `unitnames.tbl` stores
`name\0subname\0group\0` under a single index, and `stat_txt.tbl` stores a
hotkey byte followed by button text. Everything from `offset[i]` up to the next
offset in the file therefore belongs to entry `i`, which is how this tool reads
them — including bytes past the last offset, which `stat_txt.tbl` relies on.

## The text format

One entry per line, in index order. `.tbl` strings are raw bytes in a legacy
code page and are full of Windows paths, so backslash escaping would be both
lossy and miserable to hand-edit. Angle brackets are the only special
characters:

| Token | Meaning |
| --- | --- |
| `<0>` | separates the parts of a multi-part entry |
| `<N>` | the raw byte `N` (decimal 0-255) — colour codes, `<10>` newline |
| `<<` | a literal `<` |
| `<>` | nothing; marks an otherwise empty line |

Everything else is literal, so a GRP path stays readable:

```
zerg\avenger.grp
Terran Phalanx<0>Tank Mode<0>Ground Units
e<0><4>Tank Mode <1>(<3>E<1>)<10>Returns the Phalanx's weapon and mobility to normal.
```

Lines starting with `#` are comments and blank lines are ignored. An entry that
would begin with `#` or a space, or end with a space, has that character written
as `<35>` / `<32>` so no editor can quietly eat it. Output is ASCII by
construction, so re-saving the file as UTF-8 cannot corrupt it — a stray
non-ASCII character is a parse error pointing at the line and column.

## The GPTP dialect

Two tables in this repo are *already* loaded from text by the engine, not from a
`.tbl` at all: `rez/stat.txt` and `arr/sounds.txt`. CM-GPTP's `load_stat_txt`
(`hooks/limits/tbl_extender.cpp`) parses them with `collapse_control_codes` and
builds a **u32-offset** string table in memory, which is how they escape the
ceiling described below. `--dialect gptp` reads and writes that format.

The two dialects agree on what an entry is — bytes, NUL-separated — and differ
only in escaping:

| | `--dialect gptp` | tbl-text v1 |
| --- | --- | --- |
| `<N>` | byte N; **wraps** above 255 (`char()` narrowing in the C++) | byte N; over 255 is an error |
| `<<` | not special — two literal `<` | a literal `<` |
| `<>` | not special — two literal chars | nothing (empty-entry marker) |
| `\n`, `\\` | newline, one backslash | no backslash escapes |
| bytes > 127 | allowed raw | must be written `<N>` |
| `#` line | **an entry** whose text is `#…` | a comment |
| blank line | **an entry** | ignored |

That last pair is the dangerous one: the engine turns *every* line into an entry,
so a comment inserted into `stat.txt` would become a string and shift every index
after it. `--dialect gptp` therefore refuses `--annotate` and writes no banner.

Writing is unambiguous rather than byte-faithful to any hand-written file: `<` is
written `<60>` (this dialect has no `<<`, so a literal `<` before digits would be
read back as a control code), `\` is written `\\` only when the next byte would
make it an escape, and bytes above 127 are written `<N>` so output stays ASCII.
Round trips are checked against both shipped files in `test/dialect.test.ts`.

## The 64 KiB ceiling

Offsets are `u16`, so no string may start past byte 65535. `stat_txt.tbl` is
currently **26 bytes** away from that limit:

```
$ bun run tbl info ../../mpq/rez/stat_txt.tbl
  entries         1813
  size            65590 bytes (3628 header + 61962 strings)
  highest offset  65509 of 65535 (26 bytes of headroom)
```

`compile --dedupe` points identical entries at one copy of the string data,
which buys room back without changing a single string:

```
  entries         1813 (904 sharing string data)
  size            64325 bytes (3628 header + 60697 strings)
  highest offset  64244 of 65535 (1291 bytes of headroom)
```

It is off by default so that a plain recompile reproduces the original bytes.
Overflowing the ceiling is a hard error that names the offending entry and
suggests the flag.

## Library

```ts
import { Effect } from "effect"
import { Tbl, TblFiles, TblText } from "./src/index.ts"

const program = Effect.gen(function*() {
  const files = yield* TblFiles
  const table = yield* files.readTbl("images.tbl")   // ReadonlyArray<ReadonlyArray<Uint8Array>>
  yield* files.writeText("images.tbl.txt", table)
})
```

`Tbl.decode` / `Tbl.encode` and `TblText.decode` / `TblText.encode` are pure and
need no services. Failures are tagged errors — `TblBinaryError`,
`TblTextError`, `TblEncodeError` — so `Effect.catchTag` can pick them apart.

## Development

```sh
bun test          # 76 tests: byte-exact round trips over all 8 shipped .tbl files,
                  # plus the GPTP dialect against rez/stat.txt and arr/sounds.txt
bun run typecheck
```