Operational Debug Views

The operator declared debugging before the incident. The service's debug document opens one endpoint and pins each on-call engineer's key at observe. That level reads and captures but can never stop a call. Debug memory comes from its own budget, maxDebugBytes, so building a snapshot can never fail a call that would have succeeded.

# _host/s/debug.draft: who may look inside the scheduler, and how far.
u16 schemaVersion = 1
u64 maxDebugBytes = 67108864
chest[] members = [
    {
        text constructorId = "debugEndpoint"
        text key = "oncall"
        chest config = {
            text position = "listener"
            text carrier = "plaintext"
            ip address = 10.20.0.15
            u16 port = 7400
            chest[] principals = [
                {
                    u8[32] principal = [
                        41, 7, 188, 92, 13, 250, 66, 141, 9, 202, 37, 118, 75, 3, 164, 29,
                        210, 58, 97, 133, 6, 241, 84, 19, 172, 45, 230, 101, 62, 155, 8, 217]
                    text[] levels = ["observe"]
                }
            ]
            text pausePolicy = "suspendBudgets"
        }
    }
]

Ask the timer subject which timers are pending. Every subject answers the same request shape: a subject name, a key and a window. The host takes one reading between two calls, so the answer names the last committed call and never a half-run one.

# debugger -> target
text kind = "observeRequest"
u64 request = 8
text subject = "timer"
text key = "session"
chest window = { u64 start = 0, u64 count = 20 }

# target -> debugger: five session timers, where six stations run
text kind = "observation"
u64 request = 8
text subject = "timer"
text status = "ok"
u64 generation = 41
u64 total = 5
guid trace = 3f2c9a10-7b4e-4c21-9d8a-0e5f6a7b8c9d
u64 span = 418233
u64 parentSpan = 1
u64 runtime = 1
bool after = true
chest? callFirst = null
u64 content = 4
u64 size = 1912

Capture the hot path without stopping it. A capture point publishes a snapshot and lets the call continue, under a count and spacing the debugger declares. Snapshots at the line that stores each gauge reading showed Station 7's level arriving null.

# debugger -> target: at most 5 captures, at least 1 s apart
text kind = "capturePoints"
u64 request = 9
text sourceUnit = "a.gauges"
u64[] lines = [58]
u32 limit = 5
u64 spacing = 1000000000

# target -> debugger: the call kept running; its snapshot follows as content
text kind = "snapshot"
u64 request = 0
text reason = "capturePoint"
u64 calls = 418236
text cause = "published"
u64 dropped = 0
u64 refused = 0
u64 stateGeneration = 96114
u64 sceneGeneration = 0
u64 content = 5
u64 size = 22816

A pause asked at observe is refused. Stopping a call stalls every request the runtime serves, so control is a separate level the operator grants by name. The receipt says level, nothing changed, and the service kept serving.

# debugger -> target
text kind = "pause"
u64 request = 10

# target -> debugger
text kind = "receipt"
u64 request = 10
text status = "level"
u64 calls = 418240
text[] levels = ["observe"]

Reproduce the faulted call offline. The service keeps a rolling flight-recorder tape, so the engineer replays the call that faulted in a separate child with every effect denied. The replay shows each frame as it stood and reaches no live store, sensor or billing record.

# How the service runs: headless, keeping the most recent calls on tape.
tri --headless --flight-recorder 268435456 /opt/pumpsched

# On the engineer's machine: replay the tape, pinned to its digest.
tri --replay /home/oncall/pumpsched-0210.tape --tape-digest <tape-digest>
# A read the tape did not capture answers captureIncomplete;
# the replay refuses rather than guess or reach the live world.