How MaestroHub behaves when things fail
Each fault below was caused on purpose, against the whole application and real engines wherever it could be, and the result was read back afterwards. This page says what happened, what you see, and what to do.
- Covers
- Store & Forward writes to a destination
- Measured on
- MaestroHub 3.1, 9 November 2026
- Rule checked
- Missing = counted, copies within the bound
How to read this page
Each fault has one table with four rows:
- What MaestroHub does: the behaviour, as the code implements it.
- What you see: the screen, health component, metric or log line that tells you.
- What is guaranteed: missing, duplicates and order, and the measured result.
- What to do: the operator's part, often nothing.
Three words recur:
- Missing is a write the destination does not have. Every missing write must be counted: refused, evicted, kept in Failed messages, or still waiting. A missing write that is not counted is a silent loss; no fault on this page produced one.
- The in-flight bound. Store & Forward is at least once. A write that was on its way to the destination when MaestroHub crashed, restarted or moved a buffer to another replica can arrive at most one extra time, with the same
maestrohub.write-key. A cut of the link alone costs no copy on Kafka. See Delivery promise.
- Order means the order an In order output was handed its values. What the pipelines before it do to order is on Keeping readings in order end to end.
The measurements come from three kinds of run: a Kafka destination behind a proxy, cut and restored; a replay of a plant chain (OPC UA → UNS → Kafka) with cuts up to 1 hour; and fault rigs that fill, freeze or slow the data volume, kill the process, move the clock and step a three-node cluster, while a pipeline writes 50 times a second. Every whole-application run read each record back from the destination itself. The two cluster faults ran on a three-node NATS rig with the buffer's own writer and drainer, not through a destination; they say so in their tables.
The network
A broker cut shorter than the delivery timeout
| |
|---|
| What MaestroHub does | A Kafka connection is checked every second, so a cut is noticed within a few checks. A write that gets no answer within 2 seconds is buffered (the direct budget), and from then on writes are buffered without being attempted. The buffer holds, asks the connection's health check whether Kafka answers (a metadata request, no data), and sends the next record only when it does. |
| What you see | Store & Forward › Buffers: status Waiting: destination unreachable, the number waiting, and Room. The workspace Overview counts the outage as held. After 5 minutes, one No data lost: notification; one summary when it has delivered. Metrics saf_outages_open and saf_waiting_messages. |
| What is guaranteed | 0 missing, 0 duplicates, order kept. A record that was on the wire at the cut is never sent again on a timer: a resend joins the open attempt. The outage report may list it as possibly twice; it arrived once. Measured: a 2-minute cut, 7,396 readings, 0 · 0 · 0. A 1-minute cut, three times the delivery timeout: 5,000 readings, 0 · 0 · 0, first record back 1.1 s after the link returned. |
| What to do | Nothing. The buffer delivers by itself. |
A broker cut longer than the delivery timeout
| |
|---|
| What MaestroHub does | The same as above. An outage counts no attempts against any message, and the delivery timeout runs from the moment a produce starts, so a long cut never expires a held record and never blocks the buffer. |
| What you see | The Overview says things need attention: the connection failing persistently and the buffers not delivering, while every run still succeeded (a buffered write is a success). The connection list shows the connection failed after about two minutes of outage; the buffers are fine. Room counts down at the current write rate. |
| What is guaranteed | 0 missing, 0 duplicates, 0 Failed messages. Measured: a 4-minute cut at a 20-second delivery timeout, 13,424 readings, 0 · 0 · 0. Cuts of 15 minutes and 1 hour at a 5-minute timeout: 47,000 and 182,000 readings, 0 missing, 0 duplicates, the buffer never blocked, and delivery resumed by itself once the link returned. |
| What to do | Fix the network if it is yours. You do not need to raise the delivery timeout to ride out outages. Check Room if the outage may outlast it; see Sizing and protecting the buffer disk. |
A proxy that loses its brokers, or restarts
| |
|---|
| What MaestroHub does | A proxy that accepts connections with nothing behind it is a partial outage: held, not counted. The connection reconnects 1 second apart, doubling to at most 10 seconds, and is never paused by the circuit breaker. A reconnect keeps the producer, so Kafka still recognises and drops the producer's own retries. |
| What you see | The connection's Health tab: each outage, its length and the last error, which names a record whose outcome is still open. Buffers read Waiting: destination unreachable. |
| What is guaranteed | 0 missing, 0 duplicates. Measured: 90 seconds without brokers, 5,992 readings, and a 30-second proxy restart, 2,182 readings, 0 · 0 · 0, order kept. Through a proxy, delivery resumed by itself once the proxy's brokers returned, including after 15-minute and 1-hour cuts with 0 missing and 0 duplicates; on those long cuts a few dozen readings were out of input order, made by Parallel pipelines before the output (see Keeping readings in order end to end). |
| What to do | Nothing. |
A slow, lossy, flapping or half-open link
| |
|---|
| What MaestroHub does | Holds writes that get no answer in time and delivers them when answers come. A half-open link (bytes accepted, nothing answered) is an outage; Kafka learns each record's late outcome instead of resending it. |
| What you see | Buffers that come and go, Waiting while the link is bad, outage reports for each episode. |
| What is guaranteed | 0 missing, 0 duplicates, order kept. Measured, 3 to 5 minutes each: 2 s of latency each way, 9,658 writes; half-open, 9,816; down 10 s in every 20 s, 15,604; 10 % packet loss with 300 ms of delay, 10,047. All 0 · 0 · 0. |
| What to do | Nothing in MaestroHub. Fix the link. |
The message
A record the destination refuses
| |
|---|
| What MaestroHub does | A refusal no retry can fix (Kafka's INVALID_RECORD, an HTTP 400 or 422, a schema rejection) is refused for good: on the first, direct attempt the node fails with CONNECTOR_REJECTED and the record goes straight to Failed messages. A refusal a retry could fix (an HTTP 500) gets 10 counted rejections, counted only while the same connection delivers other writes, which takes 4 to 12 minutes. Then an Any order output moves the record to Failed messages and keeps delivering past it; an In order output stops at it and asks for a decision. |
| What you see | Failed messages, grouped by cause, with what the destination said: 5 failed messages, 1 cause. An In order buffer shows Blocked: needs a decision. An Action needed: notification. Metric saf_moved_to_failed_total. |
| What is guaranteed | The refused record is kept, with its payload, ready to replay. Other outputs are not affected: 40,689 readings from the other sensors in the same run, 0 · 0 · 0. The count does not grow on its own. |
| What to do | Fix the cause first, then Replay 1 to test, then Replay the other N. Use Discard only when the records themselves are wrong. On an In order buffer, decide in the decision panel. |
The process
kill -9 during an outage, and repeated during a drain
| |
|---|
| What MaestroHub does | Buffers live on disk. A write reported buffered is on the operating system before it is acknowledged, so a crash does not lose it. On start, buffers resume delivering. While the process is down its triggers accept nothing: a sender gets an error and the value never enters a buffer. |
| What you see | After the restart, the same buffers with the same backlog plus what arrived since. The outage report closes. After an unclean stop, the report shows any part it could not count (at most the last minute), and the engine log says so. |
| What is guaranteed | 0 missing. Copies only within the in-flight bound, each with the same write key. Measured: three kill -9 during one drain of 600 writes, Any order: 21 extra records for exactly the 21 writes in flight at a kill; In order: 24 extra for 24 in flight, 0 out of order. A clean stop and start with 5,688 buffered: 10,527 writes, 0 · 0 · 0. A kill -9 during a Kafka cut: 5,458 readings, 0 · 0 · 0. A power loss, unlike a crash, can lose what was acknowledged within the last second (nats.embedded.syncInterval). |
| What to do | Nothing. Restarting while buffers hold an outage is safe. A consumer that must not see a copy drops a write key it has already processed; see Delivery promises. Senders whose values were refused while MaestroHub was down should retry. |
A starved CPU
| |
|---|
| What MaestroHub does | Delivers more slowly. Nothing is given up for being slow. |
| What you see | Catching up for longer. |
| What is guaranteed | Measured with MaestroHub limited to 5 % of one core from the restore through the drain: 9,062 writes, 0 · 0 · 0; the drain completed. |
| What to do | Nothing during the incident. Size the host so it is not the bottleneck. |
The disk
The data volume fills up, under each full-buffer policy
A volume fills in steps. MaestroHub owes a different answer at each one.
| |
|---|
| What MaestroHub does | 1. The room runs out. The buffers' room follows the free space above the disk floor. When it is gone, the writing buffer's drop policy decides: Drop oldest (default) evicts its oldest messages, Reject new refuses the new write (not kept), Dead-letter new keeps it in Failed messages. 2. At the floor (5 % of the volume, between 2 and 10 GiB, unless storage.minFreeBytes is set), new values and new buffered writes are refused and counted, data past its retention is reclaimed, and Store & Forward pauses itself. It resumes by itself once free space is back above the floor plus a quarter. 3. Zero bytes, when another program takes the rest: a stream of the message store that met the full disk refuses writes. Once the volume is back above its floor, MaestroHub reopens the store from its files inside the running process (checked every 10 seconds) and Store & Forward resumes, without a restart. |
| What you see | Step 1: Dropping oldest or Rejected: buffer full on the buffer and in the Overview's evidence. Step 2: the Overview leads in red with Not kept: disk at its floor; Lost executions on the Pipelines page shows Refused at the disk floor; Store & Forward capacity shows Free, below the floor and Restarting will not free disk space; a Data loss: notification; the log line disk floor reached: Free space on … is … and the floor is …; metric storage_disk_state at 2. Step 3: the engine's health (GET /api/v1/engine/health) shows the component jetstream-store down, naming the streams, and each is logged once at ERROR; after the reopen, a WARN line says no restart needed and the component goes back up. |
| What is guaranteed | Every missing write is counted, 0 copies, and every write sent after the fault lands without a restart. A refusal is never called message too large. Measured with the volume taken to zero bytes and freed: Reject new, 18,274 writes, 13,054 missing = 574 refused by the policy + 12,480 refused at the floor. Drop oldest, 17,094 writes, 13,172 missing, every one evicted, refused or kept in Failed messages. After each fault, 577 to 999 new writes all landed. |
| What to do | Free space on the volume, or grow it. Do not restart to free space: the held records are on the same disk. If jetstream-store is still down a minute after free space is back above the floor, restart MaestroHub; that is the last resort and was not needed in any run. Then replay what went to Failed messages. Size the disk so this does not happen: Sizing and protecting the buffer disk. |
The reopen applies to the message store MaestroHub runs itself (single node). On Enterprise, the buffers live on the NATS servers; see What happens when the pool fills up.
A frozen data volume
A storage snapshot, a failing drive or a busy virtual disk can stop finishing writes without failing them.
| |
|---|
| What MaestroHub does | Only what writes to the volume waits. Reads go through their own database connections and do not queue behind a stuck write. Health is answered without asking the stuck store. A buffered write whose publish runs out of time is kept in Failed messages as Outcome unknown (possibly delivered). A run waiting behind a stuck write to an In order output fails at its deadline with ORDERED_WRITE_NOT_SENT and its write is kept in Failed messages as Waited too long for its turn. A loss that cannot be written to the ledger during the freeze is held in memory and written once the volume answers. An audit record that cannot be stored in time is logged, not kept. |
| What you see | The engine's health answers at once with the component data-volume down: the data volume has not finished a write for … : a frozen or failing disk. Sign-in answers 503: Sign-in cannot be recorded right now: MaestroHub's data volume is not finishing writes; sessions already signed in keep working. Pages that read keep answering. |
| What is guaranteed | No copies, nothing out of order, every missing write in Failed messages or counted as not kept, and every write after the thaw lands. Measured with a 90-second freeze: health and reads kept answering, and every write sent after the thaw landed. Earlier freezes of 45 seconds and 6 minutes: 0 missing, 0 copies. |
| What to do | Check the storage: a snapshot or backup freeze, a storage outage, a failing device. After the thaw, check Outcome unknown entries at the destination before you replay them. |
Two limits. Loss counts held in memory during a freeze are lost if the process stops before the volume answers. And because MaestroHub keeps its SQLite databases' write-ahead index in memory, the running server holds those databases exclusively: stop MaestroHub before you run admin-cli user commands against them (see admin-cli).
A slow disk
| |
|---|
| What MaestroHub does | Writes and the drain go at the disk's pace. Nothing is refused for being slow. |
| What you see | Slower draining; Catching up for longer. |
| What is guaranteed | Measured with 200 ms added to every write on the data volume through a whole drain: 0 missing, 0 copies, 0 out of order. |
| What to do | Nothing during the incident. Move the data to faster storage if drains matter to you. |
Booting on a small or nearly full volume
| |
|---|
| What MaestroHub does | Its own message store is given the room above the floor, and never less than 2.5 GiB, because the platform's own streams reserve about 1.9 GiB of it. So MaestroHub starts, shows the floor, and the floor holds the buffers. |
| What you see | A WARN at start: the room above the disk floor is less than the platform's own streams reserve; the store is capped at that minimum so the app starts. Then the disk-floor screens above. |
| What is guaranteed | Measured: volumes of 1, 3, 4 and 5 GiB all boot. |
| What to do | Free space or grow the volume. |
The clock
The wall clock jumps two hours, forward or back
| |
|---|
| What MaestroHub does | With the default retention (keep until delivered), nothing a buffer holds depends on the wall clock, so nothing expires, is refused or is given up. (A buffer you gave a retention age was not part of this test.) Retention clean-ups (Failed messages, the activity log, the loss evidence) wait 24 hours after a forward jump of more than 10 minutes, and the engine log says so. |
| What you see | The backlog unchanged. An access token issued on the other side of a large jump is refused: sign in again. |
| What is guaranteed | Measured with 3,981 writes held through +2 h and then −2 h: 0 lost, 0 refused, 0 Failed messages. After the link returned, 4,480 writes, 0 · 0 · 0. Run once in each direction, on an idle machine. |
| What to do | Fix the clock within 24 hours if the jump was wrong. Sign in again. |
The cluster (Enterprise)
The leader of a buffer's stream is lost
| |
|---|
| What MaestroHub does | Each buffer lives on three NATS servers. When the leader steps down or stops, another server leads and the engine keeps writing and draining. |
| What you see | Nothing on the Store & Forward pages while another server leads. The stopped server in your Kubernetes and NATS monitoring; MaestroHubNATSClusterDegraded fires only when fewer than two servers answer for 5 minutes. |
| What is guaranteed | Measured on a three-node NATS rig (not the whole application) with two leader changes and the leader stopped for 20 seconds, while a backlog of 3,000 drained and 50 writes a second arrived: 5,396 writes confirmed, 0 missing, 0 copies, 0 refused. |
| What to do | Bring the server back. |
A buffer moves to another engine replica
| |
|---|
| What MaestroHub does | The replica that takes the buffer over resumes delivery. A write that was on its way at the move may be sent once more, with the same write key. |
| What you see | Every replica shows the same health for a buffer. If the delivering replica stops reporting, the others show Stalled, naming it, until it or its successor reports. |
| What is guaranteed | At most one extra copy per write in flight. Measured in a test of the buffer's move: 8 moves with one write in flight each, 0 or 1 copy each. |
| What to do | Nothing. A consumer that must not see a copy drops a write key it has already processed. |
Running your own drills safely
Start with the built-in drill. It needs no test rig and cannot hurt other outputs:
- An outage drill cuts one output, as if its destination were down, for a time you choose. Other outputs, pipelines and the destination carry on. It ends by itself, End drill now stops it at once, it is written to the audit log, and its Drill: notifications never page anyone on call.
- Observe a real cut cuts nothing: it arms a recorder, and you cause the outage yourself, for example by stopping a proxy in a test environment.
- A platform administrator decides who may run drills on Store & Forward capacity.
For the faults on this page that a drill cannot cause (a full or frozen disk, a killed process, a clock jump), use a test installation, never production. What we learned doing it:
- Write the pass rule before the fault. Ours: missing minus counted is zero; copies are at most the writes in flight at a kill or a move; every write sent after the fault lands without a restart; health, sign-in and reads answer during it. Without numbers you check afterwards, every run looks like a pass.
- Count at the destination, not through the path you broke. Read Kafka inside the cluster, not through the proxy you cut, and read each partition to its end offset: a live topic never goes quiet.
- Cut only the destination. Drop traffic to the broker addresses only. Dropping a whole subnet can cut the published API port too, and then you are testing your own access.
- Use a separate small volume for disk faults. Fill it, freeze it or slow it from outside the MaestroHub process, and thaw it from another process: a thaw run from inside the frozen process waits forever.
- Make every payload self-delimiting, with an id per write, so a checker cannot count one body that contains another as a copy.
- Count Failed messages on every page, not the first page only.
- Sign in again after a clock jump before you call failed requests a product fault.
- Distrust a surprising number until you can explain it from first principles. 21 copies for 21 writes in flight is a result; 170,000 copies is a rig fault.
After each drill, open the outage report: the ledger must close (held = delivered + sent, not confirmed + in Failed messages + lost + waiting), and the write-key manifest lets you check every key at the destination.