Layout

+------------------------------------------------------+
|  Byte Tokens (IDs 0 - 255)                           |
|  Raw byte values 0x00 - 0xFF.                        |
|  Token ID = byte value (identity mapping).           |
+------------------------------------------------------+
|  Special Tokens (IDs 256 - 277)                      |
|  Model operation and context framing.                |
|  Never produced by raw byte encoding.                |
+------------------------------------------------------+
|  Reserved (IDs 278 - 319)                            |
|  Headroom for future domain tokens                   |
|  (e.g. <LANG>, <CURSOR> for editor domain).          |
+------------------------------------------------------+

Total vocabulary size: 320.

Byte tokens (0-255)

Identity mapping. Token ID = byte value. Encoding is type widening (uint8 -> uint16). Decoding is truncation (uint16 -> uint8). No lookup tables, no trie.

UTF-8 codepoints are naturally represented as sequences of byte tokens. Multi-byte codepoints are emitted as consecutive byte tokens:

• ASCII a (0x61) -> 1 byte token

• Cyrillic d (U+0434, bytes 0xD0 0xB4) -> 2 byte tokens

• CJK (U+4E2D, bytes 0xE4 0xB8 0xAD) -> 3 byte tokens

• Emoji (U+1F98A, bytes 0xF0 0x9F 0xA6 0x8A) -> 4 byte tokens

All shell structure is represented as raw bytes: operators (|, &&, ||, ;), quoting delimiters, flag prefixes (-, --), whitespace (space = token 32, tab = token 9), path components, and all command words. The model learns these as byte patterns from training data.

Special tokens (256-277)

22 tokens for model operation and context framing:

Token categories:

• Core tokens: <PAD>, <BOS>, <EOS>, <ATN> — sequence-level control tokens

• Frame start tokens: <CWD>, <GIT>, <HIST>, <CMD>, <ENV>, <COMP>, <QUERY>, <WORD>, <DEF>, <QUOTE>, <REF> — begin a context frame

• Subtokens (within frames): <EXIT>, <NEXT>, <POS>, <NOTE>, <IPA>, <BY> — mark substructure inside a frame

• Frame end token: <END> — terminates a frame

<NEXT> separates items within list frames (<COMP>). It replaces the space-separated encoding used in the previous tokenizer, making candidate boundaries unambiguous even when items contain spaces.

<ATN> marks the structural boundary between context frames and command input. It appears exactly once per sequence.

Encoding (text -> tokens)

function encode(text):
    tokens = []
    for pos = 0..len(text)-1:
        tokens.append(text[pos])   // byte value IS the token ID
    return tokens

That is the entire encoder. Special tokens are never produced by encoding raw text -- they are inserted programmatically by build_sequence() and calm_tokenize_frame().

Decoding (tokens -> text)

function decode(tokens):
    text = []
    for each token in tokens:
        if token < 256:
            text.append(byte(token))    // identity
        else:
            skip                        // special/frame token
    return text

Frame encoding

Simple frames (CWD, GIT, CMD, ENV) use calm_tokenize_frame():

calm_tokenize_frame(CWD, "/home/user") ->
    [CWD, '/', 'h', 'o', 'm', 'e', '/', 'u', 's', 'e', 'r', END]

Frames with subtokens (HIST with EXIT, COMP with NEXT, WORD with POS/NOTE/IPA, QUOTE with BY, REF with NOTE) embed the subtoken inline before the frame's <END>. Subtokens within a frame are optional — absent subtokens are omitted entirely:

<HIST> git diff --stat <EXIT> 0 <END>
<WORD> window <POS> n. <NOTE> OE. windowe <IPA> ˈwɪndoʊ <END>
<WORD> cat <POS> n. <END>
<DEF> An opening in the wall for the admission of light and air <END>
<QUOTE> Never say never. <END>
<QUOTE> Then to come, in spite of sorrow <BY> Milton <END>
<QUERY> what port does redis use <END>
<REF> RFC 8446, TLS 1.3 <NOTE> obsoletes RFC 5246 <END>

List frames (COMP) are constructed in Lua build_sequence() with <NEXT> separators between items. Repeated and list frames are capped at 15 items; excess entries are silently dropped (newest items are kept for history frames).

seq[#seq + 1] = COMP
for i, c in ipairs(candidates) do
    if i > 1 then seq[#seq + 1] = NEXT end
    local ctoks = core.tokenize(c)
    for _, t in ipairs(ctoks) do seq[#seq + 1] = t end
end
seq[#seq + 1] = END

Context budget

With l_max = 768, byte-level encoding has comfortable headroom:

Data structures

#define CALM_VOCAB_SIZE   320
#define CALM_BYTE_TOKENS  256
#define CALM_SPECIAL_BASE 256
#define CALM_MAX_TOKENS   1024

typedef struct {
    const char *special_names[22]; /* decode table for IDs 256-277 */
} calm_tokenizer_t;

C API

// Initialize tokenizer (populate special_names decode table)
int calm_tokenizer_init(calm_tokenizer_t *tok);

// Encode text to byte token IDs (identity mapping)
int calm_tokenize(const calm_tokenizer_t *tok,
                  const char *text, size_t text_len,
                  uint16_t *out_tokens, size_t max_tokens);

// Encode a context frame: [frame_token] [byte content] [END]
int calm_tokenize_frame(const calm_tokenizer_t *tok,
                        uint16_t frame_token,
                        const char *content, size_t content_len,
                        uint16_t *out_tokens, size_t max_tokens);

// Decode token IDs to text (bytes emitted, specials skipped)
int calm_detokenize(const calm_tokenizer_t *tok,
                    const uint16_t *tokens, size_t num_tokens,
                    char *out_text, size_t max_text);

Lua API

local calm = require("calm")

-- Tokenize raw input (byte-level identity mapping)
local tokens = calm.tokenize("git commit -m fix")
-- Returns {103, 105, 116, 32, 99, 111, ...} (byte values)

-- Build input sequence using a model's embedded template
local model = calm.load_model("~/.local/share/lilush/calm/shell.cwgt")
local seq = calm.build_sequence(model, {
    cwd = "/home/user/lilush",
    git = "main+3",
    history = {
        { cmd = "git diff --stat", exit = 0 },
        { cmd = "git status",      exit = 0 },
    },
    completions = { "commit", "checkout", "cherry-pick", "clone" },
    input = "git com",
})

-- Or build using an explicit template spec string
local pipeline = require("calm.pipeline")
local seq2 = calm.build_sequence(pipeline.TEMPLATES.shell, {
    input = "git com",
})

-- Detokenize
local text = calm.detokenize(tokens)

-- Constants
calm.PAD       -- 256
calm.BOS       -- 257
calm.EOS       -- 258
calm.ATN       -- 259
calm.CWD       -- 260
calm.GIT       -- 261
calm.HIST      -- 262
calm.EXIT      -- 263
calm.CMD       -- 264
calm.ENV       -- 265
calm.COMP      -- 266
calm.QUERY     -- 267
calm.NEXT      -- 268
calm.END_TOKEN -- 269
calm.WORD      -- 270
calm.POS       -- 271
calm.NOTE      -- 272
calm.IPA       -- 273
calm.DEF       -- 274
calm.QUOTE     -- 275
calm.BY        -- 276
calm.REF       -- 277