cosmonarchy-bw-prerelease/tools/tbl/README.md

tbl

A round-trip compiler between StarCraft: Brood War .tbl string tables and a plain-text .tbl.txt source format. TypeScript, Effect v4, Bun.

Every .tbl shipped in this repo decompiles and recompiles byte for byte identically — that is enforced by the test suite.

Usage

bun install

bun run tbl decompile ../../mpq/arr/images.tbl   # -> images.tbl.txt
$EDITOR ../../mpq/arr/images.tbl.txt
bun run tbl compile   ../../mpq/arr/images.tbl.txt   # -> images.tbl

bun run tbl verify ../../mpq/rez/stat_txt.tbl    # prove a round trip is lossless
bun run tbl info   ../../mpq/rez/stat_txt.tbl    # entry counts, offset headroom

Command	Flags
decompile <in.tbl>	-o <out>, --annotate (# [index] above each entry), --bare (no banner), --dialect
compile <in.tbl.txt>	-o <out>, --dedupe (share identical entries), --dialect
build [root]	--check (fail instead of writing), --dedupe
verify <in.tbl>	—
info <in.tbl>	—

build walks a directory for .tbl.txt sources and recompiles each one in place — the whole mpq tree in one go:

bun run build      # recompile every table under ../../mpq
bun run check      # fail if any committed .tbl is out of date

tools/hooks/pre-commit runs check so a stale .tbl cannot be committed beside an edited .tbl.txt. Install it with ln -sf ../../tools/hooks/pre-commit .git/hooks/pre-commit.

decompile defaults to <input>.txt, so images.tbl becomes images.tbl.txt and never lands on the unrelated images.txt that DatEdit uses for its name lists. compile strips that .txt back off.

The binary format

u16le  count
u16le  offset[count]     // absolute, from the start of the file
...    NUL-terminated string data

One index does not always mean one string. unitnames.tbl stores name\0subname\0group\0 under a single index, and stat_txt.tbl stores a hotkey byte followed by button text. Everything from offset[i] up to the next offset in the file therefore belongs to entry i, which is how this tool reads them — including bytes past the last offset, which stat_txt.tbl relies on.

The text format

One entry per line, in index order. .tbl strings are raw bytes in a legacy code page and are full of Windows paths, so backslash escaping would be both lossy and miserable to hand-edit. Angle brackets are the only special characters:

Token	Meaning
<0>	separates the parts of a multi-part entry
<N>	the raw byte N (decimal 0-255) — colour codes, <10> newline
<<	a literal <
<>	nothing; marks an otherwise empty line

Everything else is literal, so a GRP path stays readable:

zerg\avenger.grp
Terran Phalanx<0>Tank Mode<0>Ground Units
e<0><4>Tank Mode <1>(<3>E<1>)<10>Returns the Phalanx's weapon and mobility to normal.

Lines starting with # are comments and blank lines are ignored. An entry that would begin with # or a space, or end with a space, has that character written as <35> / <32> so no editor can quietly eat it. Output is ASCII by construction, so re-saving the file as UTF-8 cannot corrupt it — a stray non-ASCII character is a parse error pointing at the line and column.

The GPTP dialect

Two tables in this repo are already loaded from text by the engine, not from a .tbl at all: rez/stat.txt and arr/sounds.txt. CM-GPTP's load_stat_txt (hooks/limits/tbl_extender.cpp) parses them with collapse_control_codes and builds a u32-offset string table in memory, which is how they escape the ceiling described below. --dialect gptp reads and writes that format.

The two dialects agree on what an entry is — bytes, NUL-separated — and differ only in escaping:

	--dialect gptp	tbl-text v1
<N>	byte N; wraps above 255 (char() narrowing in the C++)	byte N; over 255 is an error
<<	not special — two literal <	a literal <
<>	not special — two literal chars	nothing (empty-entry marker)
\n, \\	newline, one backslash	no backslash escapes
bytes > 127	allowed raw	must be written <N>
# line	an entry whose text is #…	a comment
blank line	an entry	ignored

That last pair is the dangerous one: the engine turns every line into an entry, so a comment inserted into stat.txt would become a string and shift every index after it. --dialect gptp therefore refuses --annotate and writes no banner.

Writing is unambiguous rather than byte-faithful to any hand-written file: < is written <60> (this dialect has no <<, so a literal < before digits would be read back as a control code), \ is written \\ only when the next byte would make it an escape, and bytes above 127 are written <N> so output stays ASCII. Round trips are checked against both shipped files in test/dialect.test.ts.

The 64 KiB ceiling

Offsets are u16, so no string may start past byte 65535. stat_txt.tbl is currently 26 bytes away from that limit:

$ bun run tbl info ../../mpq/rez/stat_txt.tbl
  entries         1813
  size            65590 bytes (3628 header + 61962 strings)
  highest offset  65509 of 65535 (26 bytes of headroom)

compile --dedupe points identical entries at one copy of the string data, which buys room back without changing a single string:

  entries         1813 (904 sharing string data)
  size            64325 bytes (3628 header + 60697 strings)
  highest offset  64244 of 65535 (1291 bytes of headroom)

It is off by default so that a plain recompile reproduces the original bytes. Overflowing the ceiling is a hard error that names the offending entry and suggests the flag.

Library

import { Effect } from "effect"
import { Tbl, TblFiles, TblText } from "./src/index.ts"

const program = Effect.gen(function*() {
  const files = yield* TblFiles
  const table = yield* files.readTbl("images.tbl")   // ReadonlyArray<ReadonlyArray<Uint8Array>>
  yield* files.writeText("images.tbl.txt", table)
})

Tbl.decode / Tbl.encode and TblText.decode / TblText.encode are pure and need no services. Failures are tagged errors — TblBinaryError, TblTextError, TblEncodeError — so Effect.catchTag can pick them apart.

Development

bun test          # 76 tests: byte-exact round trips over all 8 shipped .tbl files,
                  # plus the GPTP dialect against rez/stat.txt and arr/sounds.txt
bun run typecheck