Store & Forward
Keep pipeline writes on disk while a destination is down and deliver them automatically when it recovers, so an outage does not lose data.
- Where
- Orchestrate › Store & Forward
- Default mode
- Buffer only on failure
- Disk pool
- 50 GiB shared by all buffers
- Permission
- durability:write to make changes
Overview
A pipeline's last step usually writes to something outside MaestroHub: an MQTT broker, a database, a cloud API, an S3 bucket. Those outside systems go down for reasons MaestroHub can't control: a network blip, a maintenance window, a broker restart, an expired credential.
Without protection, every write attempted during that outage is simply gone: no retry, no record, no way to replay it later. For factory-floor telemetry or billable events, that silent loss is exactly what you don't want.
Store & Forward closes that gap. It places a durable buffer between your pipeline and the destination. When the destination is reachable, messages flow straight through. When it isn't, messages pile up safely in the buffer and are delivered automatically once the destination recovers. Whether they arrive in their original order depends on the output's Delivery order. An outage becomes a pause, not a loss.
Navigation Path: Orchestrate > Store & Forward
The Overview tab while PostgreSQL is down: one buffer holds the tool-wear guard's batches, and the connection it waits for tops the list. On a healthy system every count is zero.
A healthy Store & Forward page is all zeros. Empty buffers and no failed messages mean every destination is keeping up. This page becomes valuable on the day something goes down and you watch the buffers absorb the outage instead of losing data.
The page header has a How it works button that opens a short, 5-step in-app walkthrough, including an outage simulator you can use to publish messages, take a destination offline, drop a "poison" message, and watch the buffer absorb the outage and drain in order. It's the fastest way to build intuition before a real incident.
Why we built it: the gap it closes
| Without Store & Forward | With Store & Forward |
|---|---|
| A write to a down destination is lost forever. | The write is saved on disk and delivered when the destination is back. |
| You have no record of what failed during an outage. | Every undeliverable message is visible and replayable. |
| One slow destination can stall the whole pipeline. | Each destination is isolated in its own buffer; a slow one can't block the others. |
| A restart or crash of MaestroHub loses anything in flight. | Buffers live on disk and survive restarts and crashes. |
| A bad ("poison") message can jam everything behind it. | You choose per output: set a poison message aside so good traffic keeps flowing (Any order), or hold everything behind it until you decide (In order). |
In short: Store & Forward turns "destinations are unreliable" from a data-loss problem into a routine, observable, recoverable event.
How it works
Store first, then forward
When a pipeline produces a message for an output:
- Store: the message is written to a durable buffer (the "store" step).
- Forward: a background worker continuously reads from the buffer and delivers each message to the destination (the "forward" step).
- If delivery succeeds, the message is removed from the buffer. If it fails, it stays and is retried.
How aggressively step 1 is used depends on the mode you choose per output node (see Choosing a mode below). In the default mode, MaestroHub tries the destination directly first and only starts buffering once a write actually fails, so on the "happy path" there's no extra overhead.
Buffers live on disk
Buffered messages are written to disk, not just held in memory. This is what makes the feature trustworthy: if MaestroHub restarts or crashes while a destination is down, the buffered messages are still there when it comes back up and delivery resumes automatically. Messages are only ever lost if you explicitly delete them or the underlying storage is destroyed.
One buffer per output, not one big queue
The buffer isn't a single shared queue. Every (pipeline, output node) pair gets its own buffer. This isolation matters:
- A slow or down destination only fills its buffer. Other destinations keep delivering normally.
- One destination can never reorder, starve, or block another's traffic.
- The Buffers tab is simply the list of these per-output buffers.
Example: A pipeline writes to both MQTT and Postgres. Postgres goes down. The Postgres buffer fills up while the MQTT buffer keeps delivering; the slow destination is isolated to its own row.
Only writes are buffered: never reads
Only write-type operations (publish, insert, call) are eligible for buffering. Reads and lookups never route through a buffer: buffering a read would hand the next step in your pipeline a "saved OK" acknowledgement instead of the actual query result, which would silently corrupt your data. MaestroHub enforces this automatically; there's nothing to configure.
See it in action: a database outage from start to finish
The quickest way to understand Store & Forward is to watch it absorb a real outage. Take a simple pipeline that writes one quality-check record to a PostgreSQL table every few seconds, using the default Buffer only on failure mode.
1. Normal operation: nothing to see
While the database is reachable, every record is written straight through. Rows land in the table, and the Buffers tab stays empty. This is the steady state: on the happy path Store & Forward adds no buffer and no overhead; there's simply nothing to show.
2. The destination goes down: a buffer appears and fills
Now the database goes offline (a restart, a network blip, a maintenance window). The very next write can't reach it, so Store & Forward steps in: a buffer is created for that one (pipeline → output node) pair, and every following record is stored safely on disk instead of being lost. On the Buffers tab you can watch the buffer's pending count climb with each run while the destination stays down, yet the pipeline itself keeps reporting success, because the data is already durably captured.
During the outage a buffer appears for the Store Readings write, and its waiting count climbs with every batch of ten readings. The pipeline keeps running; nothing is lost, the batches are waiting on disk.
3. The destination recovers: the buffer drains
When the database comes back, the buffer delivers its backlog automatically, without anyone touching the pipeline. Delivery starts as soon as MaestroHub's connection to the database is up again: after a short blip that is at once, but after a long outage the connection retries at a slower pace, so it can take a few minutes for the next attempt to find the database back. The buffer's pending count then falls back to zero. An empty buffer stays listed for a while: it retires itself once it has stayed empty through a few of its periodic checks, typically 10 to 20 minutes after the backlog cleared.
PostgreSQL is back: the buffer has delivered every waiting batch and its count is zero again. The outage became a pause, not a loss.
The takeaway
Count the rows in the destination table afterward and the records produced during the outage are all there; the outage shows up only as a brief delay in when they arrived, never as a gap in the data. The PostgreSQL connector declares that order matters (its Vector Upsert writes need it), so with the default Delivery order the buffer delivers in order; choose Any order on the node if the sequence of rows does not matter to you and you want the backlog to keep flowing past a refused row. That one round-trip (fill on failure, drain on recovery) is the whole promise of Store & Forward.
Choosing a mode (per output node)
Each output node decides how it uses Store & Forward. You set this in the node's configuration panel in the Pipeline Designer, and it's chosen per node: different destinations in the same pipeline can use different modes.
Where to find it: the mode selector appears on any node that writes to a destination: publishing to a broker, inserting into a database, calling an API, sending to a cloud service, and so on. Open the node in the Pipeline Designer and look for the Delivery mode option on its Parameters tab, under Advanced Options. You won't see it on nodes that only read data or that don't talk to an outside destination, since those are never buffered.
The Delivery mode choice on the guard's Store Readings node. The same option appears on any node that writes to a destination.
| Mode | What it does | Best for |
|---|---|---|
Buffer only on failure (default, on_failure) | Tries the destination directly first. Only if the write fails does it save the message to a buffer and deliver it when the destination recovers. If the write is refused for good (a retry can never succeed: the destination rejected it, or the connector refused it before sending), the node fails instead and the message goes straight to Failed messages. Lowest latency on the happy path. | Most cases. Command/control, low-volume state updates, anything where latency matters and you still want a safety net. |
Always buffer (always) | Every message goes through the buffer first and is delivered in the background. This fully decouples your pipeline's speed from the destination's speed. | High-volume telemetry where you never want the pipeline to wait on the destination. |
Do not buffer (off) | Sends directly only. If the destination is down, the message is lost and the call fails. | Best-effort telemetry where occasional loss is acceptable. Never use for billable or critical events. |
If you don't choose anything, the node uses Buffer only on failure.
Delivery order
Below the delivery modes is a Delivery order option (hidden when Do not buffer is selected). It decides what happens to the messages behind one that the destination refuses:
| Option | What it does | Best for |
|---|---|---|
| Connector default | The connector decides. Connectors that declare that order matters deliver In order; all others deliver in Any order. The option's label shows which one applies, for example Connector default (any order). | Most cases. |
| In order | If the destination refuses a message, nothing behind it is sent until an operator decides: retry, skip it, or move the backlog to Failed messages (see Failed messages). A newer value never overtakes an older one in the buffer. | Setpoints and commands. |
| Any order (fastest) | A refused message retries behind newer ones, and newer messages keep flowing. An older value can land after a newer one. | Timestamped readings. Wrong for commands. |
A connector's default is the strictest order any of its buffered writes declares: a connector with even one write that needs order (a message stream such as Kafka or Azure Event Hubs, a keyed store such as MongoDB or PostgreSQL, a TCP stream) delivers In order by default; connectors whose writes are all unordered, such as MQTT, REST, S3 and SAP IDoc, deliver in Any order. The node's Connector default label shows which one applies. Choosing Any order on a connector that asks for in-order delivery shows a warning.
Delivery order keeps the order in which the pipeline's runs handed messages to the buffer. With the pipeline's default Run order, Parallel, runs overlap and can hand them over in a different order than their inputs arrived; set Run order to One at a time, in arrival order when the destination must receive values in input order (see Output order).
The Delivery order option on the Store Readings node. The connector default comes from the connector; for PostgreSQL it is in order, so the label reads Connector default (in order).
After a failure in the default mode, that output keeps buffering subsequent messages (instead of trying directly again) until its buffer fully drains. This is deliberate: a direct write never overtakes messages still waiting in the buffer. The order in which the buffer delivers those messages follows the Delivery order above.
When and how it delivers: retries, failed messages, and the brakes
When a destination misbehaves, three mechanisms decide what happens next. Together they guarantee every message has a predictable landing place: delivered, failed, or paused for an operator.
Retries
- Temporary errors (timeouts,
503s, connection refused) are retried automatically with backoff: increasing waits between attempts, up to a per-buffer cap (10 attempts by default). - Rate-limited responses wait exactly as long as the destination asks before retrying, without burning an attempt.
- A connection that can't take writes right now (stopped, suspended, still connecting, reconnecting, or paused for workspace maintenance) doesn't use up attempts either. The buffer holds every message, sends nothing while the connection is stopped, and delivers as soon as it's back, however long that takes.
This handles the most common case: a destination that's briefly unavailable and then comes back.
Failed messages (the triage inbox)
Some messages can't be saved by retrying:
- Permanent failures (shown as Refused for good):
4xxerrors, schema/validation rejections, contract mismatches ("poison" data), and writes the connector refuses before sending anything, such as data that holds no rows. - Retry-exhausted messages that hit the attempt cap.
These land in Failed messages (internally a dead-letter queue, or DLQ) with the failure reason attached.
A permanent failure on the first, direct attempt never enters the buffer at all: sending it again could only fail the same way (and would deliver a message twice where the destination did act on it). The output node fails with the reason, code CONNECTOR_REJECTED, and is not retried, so the run shows the problem instead of turning green. The reason is either the destination's own error or, when the connector refused before sending anything (for example a write whose data holds no rows), the connector's explanation, which says nothing was sent. The message goes straight to Failed messages: replay it once you fix the cause, or discard it when the message itself is the problem. A failure that a retry can fix (the destination is down, slow or throttling) is buffered and delivered later, exactly as above.
What happens to the messages queued behind one depends on the output's Delivery order:
- Any order: the failed message is set aside here and delivery keeps flowing past it.
- In order: nothing behind the refused message is sent. The buffer becomes Blocked and waits for an operator: on the Buffers tab, Retry sends the stuck message again and then the rest in order; Skip stuck message moves only that message to Failed messages and resumes delivery; Move to Failed messages moves the stuck message and everything behind it.
From the Failed messages tab an operator can:
- Replay a message (single or in bulk) after fixing the underlying cause. A replayed message goes back into its buffer and leaves the Failed messages list; the buffer delivers it like any other. If it fails again it comes back as a new entry that shows how many times it has been retried.
- Purge messages that are genuinely unwanted (single or in bulk).
Failed messages are kept for 30 days by default, with a cap of 10,000 entries per buffer so a flood can't fill the disk.
When messages are moved here automatically, or a buffer becomes stalled, blocked or starved, MaestroHub sends a Delivery health notification, which you can route to Slack, Teams or a webhook on the Notifications page.
Failed messages after a migration made a column mandatory during an outage: each batch the database refused is here once, marked permanent, with the database's own error. Fix the cause, then replay them.
The brakes
When you need to intervene, you have manual controls:
Per buffer:
- Pause: stop accepting new messages into this buffer; keep delivering what's already queued.
- Drain: stop accepting new messages and deliver the backlog, then retire the buffer.
- Purge: drop the buffer's contents.
Instance-wide:
- Emergency pause (the kill switch): stops all new buffering across the instance, in every workspace, while letting already-queued messages continue draining. Use it when a destination is making things worse and you need everything to stop accepting new writes immediately. Every toggle is recorded (who, when), and replaying failed messages is blocked until you resume.
Emergency pause engaged on the Overview tab: no new message is buffered anywhere on this instance until someone resumes it.
Every state change a buffer goes through is audited. You can always see who paused, drained, purged, or resumed, and when. (See Where to see what happened below.)
When Store & Forward itself can't buffer
Very rarely, the safety net itself can't accept a message, meaning the message was neither delivered nor stored. If this happens, the Overview tab shows a prominent amber banner: "Store & Forward could not buffer N message(s) in the last hour." The banner breaks down the cause so you know what to fix:
| Cause shown | What it means | What to do |
|---|---|---|
| Capacity exhausted | The disk pool ran out of room. | Raise the capacity pool, or shrink an existing reservation. The banner links straight to Capacity & Limits. |
| Substrate error | The underlying buffer broker refused the write. | Check the broker's health and the engine logs. |
| Buffer plumbing unavailable | A required Store & Forward component wasn't available. | Check the engine startup logs. |
| Internal error | An unexpected engine error while buffering. | Check the engine logs for the root cause. |
This banner is the one signal that data may actually have been lost, so it's worth acting on immediately. On a healthy system it never appears.
Capacity: where the buffers' space comes from
All buffers draw from one instance-wide disk pool. You don't have to size each buffer by hand; most of this is automatic, and there's only one number most operators ever touch.
The instance-wide pool (the one number you control)
There's a single total disk budget shared by every buffer on the instance, in every workspace. It defaults to 50 GiB and is the main thing an operator decides. You can change it any time from the Capacity & Limits tab; the Overview tab shows a gauge of how much of it is currently in use. For most installations the default is plenty and you'll never need to change it.
Each buffer's size is automatic
You do not have to assign a size to each buffer. When an output first needs to buffer, MaestroHub creates that buffer automatically with a per-buffer cap of 256 MiB by default. Every buffer draws the space it actually uses from the shared pool as messages arrive, and gives it back as they drain. So in a typical setup, the only capacity decision is the total pool above; everything below it takes care of itself.
Optional: reserve space for a critical destination (guarantees)
By default no buffer has space reserved for it; they all compete fairly for the shared pool. If you have a destination you never want to be squeezed during a busy outage, you can give its buffer a guarantee: a slice of the pool reserved just for it, that no other buffer can take. This is optional and set per buffer; you only do it for the destinations that matter most.
The leftover is a shared "burst" reserve
Whatever isn't reserved by guarantees is a shared burst pool. Any buffer can borrow from it during a bad outage (so a single destination going down can use far more than its everyday footprint) and release it once it drains. The system computes this automatically; there's nothing to set.
Optional: per-workspace quota (several workspaces on one instance)
If several workspaces share one MaestroHub instance, you can cap how much of the global pool any single workspace's buffers may reserve. This is the Your workspace's quota card in the Capacity & Limits tab, where you can set, edit, or clear a quota. With no quota set, a workspace is bounded only by the global pool minus what other workspaces have reserved. A quota only limits reservations (guarantees); the shared burst pool stays best-effort across workspaces. Most instances with a single workspace never need this.
The total pool is read from configuration at startup (and editable any time afterward). Each buffer's size is decided automatically when the buffer is first created, using its guarantee if you set one, otherwise the 256 MiB default. Burst availability is recalculated continuously. The only values an operator sets are the total pool and, optionally, per-buffer guarantees.
The Capacity & Limits tab: the instance-wide disk pool with the portion in use, per-buffer guarantees, the per-workspace quota, and the recent activity log.
What happens when a buffer fills up
Nothing grows without a limit you chose. When a buffer reaches its capacity, its drop policy decides what happens, and no loss is ever silent:
| Drop policy | Behavior |
|---|---|
| Drop oldest (default) | Evict the oldest message to make room for the new one. |
| Reject new | Refuse new messages until space frees up. |
| Dead-letter new | Send new messages straight to Failed messages instead of dropping them. |
The four tabs at a glance
| Tab | What it's for |
|---|---|
| Overview | "Is anything stuck?": the one-glance answer. Emergency pause state, buffers grouped by status, capacity gauge, failed-message count, and the destinations with the most pending messages. |
| Failed messages | The triage inbox. Filter by reason, replay after fixing the cause, or purge poison batches, single or in bulk. |
| Buffers | Every per-output buffer with its depth (pending messages/bytes), age of the oldest message, drain rate, and per-buffer actions (pause / drain / purge / replay). |
| Capacity & Limits | Size the instance-wide disk pool, set per-buffer guarantees, manage a per-workspace quota, find the emergency pause, and review the recent Activity (audit) log. |
Viewing every tab is open to anyone with access to the page. Making changes (pausing, draining, purging, replaying, editing capacity or quotas, toggling the emergency pause) requires the durability:write permission. Without it the page is read-only: the action menus and buttons are hidden, but you can still see the full picture and the activity log.
Buffer states and what they mean
A buffer moves through a small set of lifecycle states. Most of the time you'll only see Healthy. The labels below are exactly what the UI shows (the technical name is in parentheses).
| Status (UI label) | Meaning |
|---|---|
| Healthy (active) | Accepting and delivering messages normally. |
| Blocked (blocked) | The buffer delivers In order and the message at the front of the queue could not be delivered: it was refused for good (by the destination, or by the connector before sending anything), or its retries ran out. Nothing behind it is sent until you choose Retry, Skip stuck message, or Move to Failed messages. New messages still queue behind it. |
| Paused (paused) | You stopped accepting new messages; the queued backlog still drains. |
| Finishing delivery (draining) | No longer accepting new messages, delivering what's queued, then the buffer goes away. |
| Source missing (orphaned) | The pipeline or connection was deleted. The buffer isn't silently discarded; it's parked for a day so you can deliberately recover or discard it. |
| Held for review (dead-letter) | Couldn't deliver after the orphan window expired; remaining messages were moved to Failed messages. |
| Removed (dead) | The buffer was removed. Its history is kept in the audit log. |
Health (operational hints)
Separate from the lifecycle state, a healthy buffer can still be operationally stuck. When available, the Buffers tab shows a Health badge: click it for the reason, when it started, and the last successful delivery:
| Health (UI label) | Meaning |
|---|---|
| Healthy | Messages flowing: delivery completed recently. |
| Idle | Nothing queued and the destination is fine: a quiet topic, no action needed. |
| Blocked | The destination connection isn't delivering. Fix the named connection to unstick it. |
| Stalled | Messages are queued and the destination looks healthy, but nothing is draining, so escalate. |
| No input (starved) | Nothing is arriving from upstream. The upstream connection is likely unhealthy. |
| Measuring… (unknown) | Not enough signal observed yet; usually clears within ~30 seconds. |
Where to see what happened: the activity log
Every consequential action is recorded: each emergency-pause flip, capacity edit, buffer state change, and Failed-messages purge. The Capacity & Limits tab has an Activity card showing the most recent events (when, who (the actor), the action, the result (success / denied / failed), and details), with a View full audit log link for the complete history. This is read-only and visible to everyone, so anyone can see what changed without needing write access.
Edge cases worth knowing
- MaestroHub restarts during an outage. Buffered messages are on disk; they're still there after the restart and delivery resumes automatically.
- You stop or suspend the destination connection. Writes keep being buffered and none are sent or failed while it's stopped; they're delivered when you start or resume it.
- A poison message in the middle of the queue. With Any order, it's moved to Failed messages and good traffic behind it keeps delivering; fix the cause, then replay it. With In order, the buffer becomes Blocked and nothing behind the message is sent until you retry, skip it, or move the backlog to Failed messages.
- The destination is rate-limiting you. MaestroHub honors the requested delay instead of hammering the destination or wasting retry attempts.
- A pipeline or connection is deleted while its buffer still has messages. The buffer becomes Orphaned and is kept for a day so the data isn't lost by accident; recover or purge it deliberately.
- A buffer fills to capacity. Its drop policy decides the outcome (drop oldest / reject new / dead-letter new). The loss is never silent; it's the policy you chose, and it's visible.
- Emergency pause is on and you try to replay failed messages. Replays are blocked until you resume, so you don't reintroduce traffic into a system you just told to stop.
- You pick "Do not buffer" (
off). There's no safety net: a write to a down destination is lost and the call fails. Reserve this for best-effort telemetry only. - Reads are never buffered. Even in "Always buffer" mode, lookups/reads always go directly to the destination so downstream steps get the real result.
- A replay was interrupted by a crash. If the engine crashed mid-retry, the message may have reached the destination or may not. The detail view flags this: check the destination before retrying again. Retrying within 10 minutes of the interrupted attempt never queues a second copy; after that, a blind retry can create a duplicate.
- A message was retried too many times. Further automatic retries are blocked to avoid loops; the entry is flagged in Failed messages. Fix the root cause and retry deliberately, or delete it.
- Someone else is already replaying an entry. While a replay is in progress the Retry button is disabled for that entry; if it appears stuck for more than a few minutes, it clears on the next engine restart.
- Bulk retry / delete in Failed messages. Bulk actions handle up to 100 entries per request; select fewer if you have more, and repeat. The result is summarized (replayed / skipped / failed).
- Store & Forward couldn't buffer at all. See When Store & Forward itself can't buffer: this is the one case where data can actually be lost, and the Overview banner tells you the cause.
Defaults reference
| Setting | Default |
|---|---|
| Mode (per output node) | Buffer only on failure (on_failure) |
| Delivery order (per output node) | Connector default |
| Storage | On disk (survives restarts) |
| Instance-wide disk pool | 50 GiB |
| Per-buffer size cap (when no guarantee set) | 256 MiB |
| Per-buffer guarantee | None (0, optional, set per buffer) |
| Retry attempts before failing | 10 |
| Failed-message retention | 30 days |
| Failed-message cap per buffer | 10,000 entries |
| Drop policy when a buffer is full | Drop oldest |
| Orphaned-buffer retention | 24 hours |
Related
- Pipelines: where output nodes (and their Store & Forward mode) are configured.
- Dependencies: see which pipelines write to which connections.