Wiki-DB -- Self-Describing MNEME Wiki Databases

Overview

A convention for MNEME databases that makes them self-describing, portable wiki files. A single .mneme file carries its content, search indexes, browsing indexes, presentation style, and optionally an embedding model.

Convention version

The current convention version is 1.

A viewer must check the version key in the manifest keyspace before proceeding. If the database version is greater than the version the viewer supports, it must refuse to open the database with a clear error.

Manifest keyspace: __wiki

A conforming wiki-db has a keyspace named __wiki containing well-known keys that describe the rest of the database. Each manifest field is a separate key.

The __wiki keyspace name is reserved. Applications must not use it for any other purpose.

Required keys

KeyValue typeDescription
versionintegerConvention version (currently 1)
titlestringHuman-readable wiki title
content_keyspacestringName of the FT keyspace holding readable content
content_formatstring"markdown" or "plain"

Optional keys

KeyValue typeDescription
descriptionstringShort description of the wiki's content
meta_keyspacestringName of the KV keyspace holding entry metadata
indexesstring (JSON)Array of browsing index descriptors
entry_pointstring (JSON)View directive for initial landing
tssstring (JSON)TSS stylesheet for rendering content
vec_keyspacestringName of the KV keyspace holding vector embeddings
vec_metricstringDistance metric: "cosine", "dot", or "l2"
modelbytesCWGT blob -- embedded CALM model for vector search
collectionsstring (JSON)Array of additional content collection descriptors

Content organization

Content keyspace

The primary content lives in an FT keyspace (full-text indexed). Each entry is a document keyed by its entry ID. The content format is declared in the manifest: "markdown" (preferred) or "plain".

local ft = db:ft_keyspace("content")
ft:put("rfc-790", "# RFC 790\n\nAssigned Numbers\n\n...")

Metadata keyspace

An optional KV keyspace holding per-entry metadata as JSON strings. Each key matches the corresponding entry ID in the content keyspace.

local meta = db:keyspace("meta")
meta:set("rfc-790", json.encode({
    title = "Assigned Numbers",
    authors = {"J. Postel"},
    date = "1981-09",
    status = "Historic"
}))

The metadata schema is not prescribed by the convention, applications store whatever fields are useful for their domain. The viewer reads available fields and adapts its display accordingly.

Shared key convention

The same key identifies an entry across all keyspaces. Entry rfc-790 in the content FT keyspace corresponds to rfc-790 in the metadata KV keyspace and serves as a prefix for related vectors.

Browsing indexes

Browsing indexes are sorted sets in the metadata keyspace that provide ordered views over the entries. Each index is described in the indexes manifest key as a JSON array.

Index descriptor

[
    {"keyspace": "meta", "set": "by_date",  "label": "By date",  "order": "desc"},
    {"keyspace": "meta", "set": "by_pages", "label": "By size",  "order": "desc"}
]
FieldTypeDescription
keyspacestringKeyspace containing the sorted set (defaults to collection's meta)
setstringSorted set name
labelstringHuman-readable label for the viewer UI
orderstringDefault display order: "asc" or "desc"
collectionstringOptional: name of the collection this index belongs to

Each sorted set maps scores to entry IDs:

local zs = meta:sorted_set("by_date")
zs:add(198109, "rfc-790", 202301, "rfc-9293")

Entry point

The entry_point manifest key is a JSON object that tells the viewer what to show when a wiki-db is first opened.

Browse entry point

{"mode": "browse", "index": "by_date", "order": "desc"}

Opens the specified browsing index. The index value must match a set name from the indexes array.

Search entry point

{"mode": "search"}

Presents the user with a search prompt.

If entry_point is absent, the viewer should default to search mode when no indexes are available, or to the first declared index otherwise.

Collections

A wiki-db may contain additional content collections beyond the primary one. Each collection is a self-contained FT keyspace with its own optional metadata keyspace, enabling a single database to hold different types of content (e.g., dictionary entries and literary quotations).

Collection descriptor

The collections manifest key is a JSON array of collection objects:

[
    {
        "name": "quotes",
        "label": "Quotes",
        "content_keyspace": "quotes",
        "content_format": "markdown",
        "meta_keyspace": "quotes_meta"
    }
]
FieldTypeRequiredDescription
namestringyesUnique collection identifier
labelstringnoHuman-readable label (defaults to name)
content_keyspacestringyesFT keyspace holding collection content
content_formatstringno"markdown" or "plain" (defaults to primary)
meta_keyspacestringnoKV keyspace for collection metadata

The primary collection is always defined by the top-level content_keyspace, content_format, and meta_keyspace fields. The collections field defines additional collections only.

Collection-aware indexes

Index descriptors gain an optional collection field that binds the index to a specific collection:

[
    {"set": "by_alpha", "label": "Alphabetical", "order": "asc"},
    {"set": "by_author", "label": "By author", "order": "asc", "collection": "quotes"}
]

When collection is absent, the index belongs to the primary collection. When present, the viewer uses the named collection's meta keyspace for the sorted set and title lookups, and the collection's FT keyspace for content retrieval.

Viewer behavior

Browsing an index sets the viewer's current collection. Subsequent view and search commands operate on the current collection. The search -c <name> flag explicitly targets a collection.

Vector search

When vec_keyspace is declared, the wiki-db supports semantic search over vector embeddings.

Vector keyspace

Embeddings are stored in a dedicated KV keyspace using MNEME's native vector type. Vectors may use compound keys of the form {entry_id}:{chunk_idx} when multiple embeddings represent a single entry (e.g., different sections of a long document).

local vec = db:keyspace("vec")
vec:vset("rfc-790:000", embedding_abstract)
vec:vset("rfc-790:001", embedding_section1)

The vec_metric manifest key declares the distance metric used for search: "cosine" (default if absent), "dot", or "l2".

Embedded CALM model

When the model key is present in __wiki, it contains a CWGT binary blob -- a complete CALM model that the viewer loads into memory for on-the-fly query embedding.

This makes the wiki-db fully portable: vector search works without any external model files. Even the largest CALM model (small, ~24MB) is negligible relative to typical wiki-db sizes.

local wiki = db:keyspace("__wiki")
local model_blob = wiki:get("model")
-- Load model from blob and use for query embedding

When vec_keyspace is declared but model is absent, the viewer may fall back to FT-only search, or load a model from a known system path.

Presentation

TSS stylesheet

The optional tss manifest key holds a TSS stylesheet serialized as JSON. The viewer deserializes it to a Lua table and passes it to the pager for rendering.

local wiki = db:keyspace("__wiki")
local tss_json = wiki:get("tss")
if tss_json then
    local rss = json.decode(tss_json)
    local style = tss.new(rss)
    pager:set_style(style)
end

If tss is absent, the viewer uses its default stylesheet.

Viewers

A wiki-db viewer is any application that opens a conforming .mneme file, reads the __wiki manifest, and presents the content to the user.

Graceful degradation

A conforming viewer should adapt to what the manifest advertises rather than requiring every optional feature to be present:

ConditionExpected viewer behavior
No indexesBrowse unavailable; search-only mode
No vec_keyspaceSearch uses BM25 only, no hybrid ranking
No modelVector search unavailable even if vectors exist
No tssViewer's default stylesheet used for rendering
No meta_keyspaceSearch results show entry IDs only, no enrichment
No entry_pointDefault to first index if available, else search
No collectionsOnly the primary collection is available

Lilush Shell Wiki mode

The Lilush Shell includes a built-in wiki-db viewer as one of its shell modes, activated with F3.

It provides a command-driven interface with tab completion, renders content through the shell's pager (with markdown support and optional custom TSS from the database), and performs hybrid search (BM25 + vector via RRF) when the database includes an embedded CALM model. Type help inside wiki mode for a quick command reference.

Database selection

The wiki command selects which database to view:

InvocationBehavior
wikiList available databases under ~/.local/share/lilush/wiki/
wiki <name>Open/switch to a named database from the default directory
wiki /path/to/db.mnemeOpen/switch to a database at an arbitrary path

Database names support tab completion over .mneme files in the default directory.

Commands

CommandDescription
search [-n N] [-c coll] <query>Full-text search; hybrid when available; -c targets a collection
browse [index] [-o N] [-n N]Browse entries from a sorted-set index with pagination
view <key>Render an entry in the pager with the wiki's TSS style
infoShow wiki metadata: title, description, available features
helpShow command usage summary

Example: creating a conforming wiki-db

local mneme = require("mneme")
local json  = require("std.json")

local db = mneme.open("my_wiki.mneme")

-- Set up manifest
local wiki = db:keyspace("__wiki")
wiki:set_integer("version", 1)
wiki:set("title", "My Knowledge Base")
wiki:set("description", "Personal notes and references")
wiki:set("content_keyspace", "content")
wiki:set("content_format", "markdown")
wiki:set("meta_keyspace", "meta")
wiki:set("indexes", json.encode({
    {keyspace = "meta", set = "by_date", label = "By date", order = "desc"}
}))
wiki:set("entry_point", json.encode({mode = "browse", index = "by_date"}))

-- Populate content
local ft = db:ft_keyspace("content")
ft:put("hello-world", "# Hello World\n\nThis is the first entry.")

-- Populate metadata
local meta = db:keyspace("meta")
meta:set("hello-world", json.encode({
    title = "Hello World",
    date = "2026-04-01"
}))

-- Create browsing index
local by_date = meta:sorted_set("by_date")
by_date:add(20260401, "hello-world")

db:close()

Reserved names

NameScopePurpose
__wikiKeyspaceManifest keyspace; must not be used for other purposes