Engineering reference / System overview

Persistence

From a typed value to bytes that survive

Persistence in MTK is one path from a typed Trident value to bytes that survive a crash. The value is encoded once as Rune, stored by MerDB with any large body kept apart by key, and written to checked media through MerDurable. Each write reports what it survived, and committed changes leave in order through the change feed.

Worked example

A field tablet loses power mid-save

A gas utility’s survey crew logs valve inspections on rugged tablets at remote well pads. Each inspection is a 200-byte row plus a 14 MB photo, saved over a cellular link to the district server’s store. A tablet battery can die at any byte of a save. The office work-order service must receive every saved inspection, in order, and never one whose photo is missing.

The photo gets its own trove, with an external column for its bytes, and the inspection row holds a key to it. ack = durable makes every inspection write wait for the server’s durability barrier unless a call asks for less, and deletion = denied keeps inspections from ever being deleted.

# _db/troves/photo.tri
trove photo {
    text camera = ""
    timepoint<unix> takenAt = now()
    buffer? body = null with { storageClass = external }
}

# _db/troves/inspection.tri
trove inspection {
    ack = durable
    deletion = denied
    text site = ""
    text valve = ""
    text verdict = "open"
    ref photo? image = null
    index bySite = site
}

The tablet saves the inspection with one call. The photo value is not stored yet, so the store saves it in the same commit as the row that references it. insertDurable returns only after that commit has passed the server’s durability barrier.

text saveInspection(text site, text valve, text verdict, buffer jpeg) {
    photo shot = { camera = "tablet-07", body = jpeg }
    transaction {
        inspection.insertDurable({ site = site, valve = valve, verdict = verdict, image = shot })
    } else (f) {
        return "not saved; kept on the tablet for another try"
    }
    return "saved"
}

Power fails while the photo is still uploading. Before the save left the tablet, its runtime stored a sealed receipt for the transaction: the pending record. After reboot it asks the store to resolve that receipt. The answer is applied, aborted, committed with parts still applying, or unresolved, and a missing decision never counts as an abort.

The office service follows the inspection change feed. A change reaches the feed only after it meets the trove’s acknowledgment tier, durable here. The office never opens a work order for an inspection a failure could still erase.

u64 openWorkOrders(u64 savedThrough) {
    resource<t.db.subscription> feed = t.db.subscribe("inspection", savedThrough)
    chest page = t.db.poll("inspection", feed, 32)
    chest[] changes = page.events
    foreach (change in changes) {
        u64 id = change.id
        chest row = t.db.get("inspection", id)
        chest doc = row.document
        text verdict = doc.verdict
        if (verdict == "leak") {
            t.log.info("open a work order for inspection " + text(id))
        }
    }
    u64 through = page.through
    t.db.ack("inspection", feed, through)
    return through
}

What happened

Tablet 07 died 9 MB into a photo upload at Pad 14. Its receipt showed no commit decision, so after a recharge it recorded an explicit abort and saved again, and the office received the inspection once. When the district server lost power later that week, its store reopened at the last acknowledged state, or one write past it, with nothing half-written.

System diagram

Accepted update through recovery

The pending record closes the gap between accepting a change and making it durable.

Typed update part P-204 installed
Pending record local durable intent
Commit row + image object key
Change stream sequence 44,018 emitted

Inside the system

How the parts work together

Follow the data, decisions, and access boundaries. Each diagram explains a concept used by applications.

Map 01

One encode, several owners

Application, schema, database, byte store, and durability each own one step.

Application value typed intent
Schema approved shape
Rune one encoding
MerDB row ownership
MerFS large bytes

Map 02

Choose the data form

Typed binary is the default; JSON enters only through a narrow explicit boundary.

Typed value known schema
Rune normal path
External JSON edge input only
Validation map to typed value

Map 03

Row and large object

A small fixed key joins a typed row to separately stored media.

Event row time + sensor + status
Object key stable reference
Media bytes image or tile
Read join only when needed

Map 04

Pending state survives failover

Accepted work becomes durable intent before a new owner may continue it.

Edit accepted dirty state
Pending record durable intent
Owner changes failover
Recovery commit or named refusal

Map 05

Ordered durability

Commit records, byte storage, and the change cursor advance in a defined order.

Pending write validated
Store bytes durable object
Commit row accepted state
Advance cursor consumer-visible change

Key parts

What the system does

These are the main boundaries, inputs, outputs, and failure rules. The examples show a specific use of each part.

One encode

A value is encoded once, as Rune, and those bytes are the stored row and the response body. No object mapper, serializer or text conversion sits between the program and the disk.

For example

The inspection record the tablet builds is byte for byte the record the office reads back.

Rows and large objects

A row stays small and holds a key to its large object. The large body lives in an external column, stored in chunks beside its own row, and a large upload commits as one write. A reader sees the old body or the new one, never a prefix.

For example

A 14 MB valve photo sits beside a 200-byte inspection row, and the work-order list reads only the rows.

Ordered durability

Each write states what it must survive. fast returns when the volume primary holds it, replicated when a standby holds it too, and durable after the medium’s barrier. Every stored value is stamped with the tier it reached and what confirmed it, and the change feed releases a change only at its trove’s tier.

For example

An inspection written with insertDurable returns after the server’s disk barrier, and only then does the office feed show it.

Code

Persistence in code

Each example is short and complete enough to read on its own. Hover a dotted word for its definition.

Trident

Choose what each write survives

The tier is part of the call. A 10 Hz sample can live with losing the last few readings; an overpressure alarm cannot, so it waits for the durability barrier.

# _db/troves/pressureReading.tri
trove pressureReading {
    text site = ""
    f64 kpa = 0.0
    bool alarm = false
}

# a/pressure.tri
text record(text site, f64 kpa) {
    if (kpa > 550.0) {
        # an overpressure alarm must survive a power cut
        pressureReading.insertDurable({ site = site, kpa = kpa, alarm = true })
        return "alarm stored"
    }
    # a routine 10 Hz sample: the primary's memory is enough
    pressureReading.insertFast({ site = site, kpa = kpa })
    return "sample stored"
}

Trident

One encoding from program to row

The document the program built is the document the store holds. Reading it back gives the same bytes, so nothing between the two can rename a field or round a number.

text main(text[] argv) {
    chest reading = { text site = "Pad 14", text valve = "V-12", text verdict = "leak" }
    chest written = t.db.put("inspection", 0, reading)
    u64 id = written.id
    chest row = t.db.get("inspection", id)
    chest stored = row.document
    assert (t.chest.encode(stored) == t.chest.encode(reading), "the stored row is the same bytes")
    return "inspection " + text(id)
}

Trident

Encrypt a column before it leaves the runtime

The writing runtime encrypts gateCode before the value leaves the process, so the store, its disks and its backups hold only ciphertext. A read returns ciphertext, and open() turns it into a secret under the column’s key.

# _db/troves/siteAccess.tri
trove siteAccess {
    text site = ""
    text gateCode = "" with { encrypted, pii = true }
    unique index bySite = site
}

# a/access.tri
text recordGate(text site, secret text code) {
    siteAccess.insert({ site = site, gateCode = code })
    return "stored"
}

Trident

Hear about a fast write that was lost

insertYolo returns before the store answers. If the store later loses or refuses the write, the runtime calls on writeLost with a reason such as conflict, check or backpressure. With no handler, the loss is still counted in the store’s metrics.

text sample(text site, f64 kpa) {
    # returns at once; nothing waits for the store
    pressureReading.insertYolo({ site = site, kpa = kpa })
    return "sent"
}

on writeLost(const w) {
    t.log.warn("a pressure sample was not stored")
}

Shell

The same program on any store

The store is chosen at launch, never in code. No call refuses or compiles differently because of its store. A durable write on the memory store is stamped durable (memory), so nobody mistakes it for a disk write.

# a laptop test: the store lives in memory and ends with the process
tricli --store memory survey.tri

# a field server: a disk store whose barrier confirms durable writes
tricli --store disk /srv/survey/store survey.tri

Uses

Example uses

Field data captured on unreliable power
Sensor and inspection archives with large media
Work-order services driven by committed changes

Review questions

What the team must decide

Which writes must survive a power cut, and which can trade that for speed

Which large bodies should live apart from their rows

What a client does with a save whose outcome it never heard

Persistence

A field tablet loses power mid-save

Accepted update through recovery

How the parts work together

What the system does

One encode

Rows and large objects

Ordered durability

Persistence in code

Choose what each write survives

One encoding from program to row

Encrypt a column before it leaves the runtime

Hear about a fast write that was lost

The same program on any store

Example uses

What the team must decide

See how this fits the wider platform