Skip to main content
Version: 3.0 (next)

MQTT broker runbooks

One entry per broker alert, and step-by-step guides for the situations operators meet with a clustered broker.

For
Whoever runs the Kubernetes deployment
Alerts
Helm chart, alerts.enabled
Alert groups
maestrohub.uns-ingest · maestrohub.uns-broker-cluster

The Kubernetes chart ships a PrometheusRule (alerts.enabled) with ten alerts about the built-in MQTT broker: three in the group maestrohub.uns-ingest (values devices publish, on their way to the historian) and seven in maestrohub.uns-broker-cluster (the UNS replicas acting as one broker). Each alert's runbook_url points at its entry below.

How the cluster is built, and what survives a failure, is on How the broker cluster behaves. Backing up its state is on Back up and restore the broker's NATS state.

Before you start​

The commands below assume the chart's defaults. Replace them with yours:

PlaceholderDefault
<app-ns> (UNS pods)maestrohub
<deps-ns> (NATS)maestrohub-deps
<release>maestrohub

Where to look.

  • Metrics. Every counter below is exported with the maestrohub_ prefix and the scrape target's namespace and pod labels. Each UNS replica counts its own.

  • Logs. Every broker line starts with the component it comes from: broker cluster:, mesh: or embedded broker:. Read them per replica:

    kubectl -n <app-ns> logs <release>-uns-0 | grep -E 'broker cluster:|mesh:|embedded broker:'

  • Readiness. Each UNS pod answers GET /health/ready on port 8080: 200 when it serves, 503 with "status": "not_ready" and a reason when it does not. A broker reason starts with broker-cluster-store:.

    kubectl -n <app-ns> port-forward pod/<release>-uns-0 8080:8080
    curl -s localhost:8080/health/ready
    curl -s localhost:8080/health      # the mqtt-broker component, with its message

  • Status APIs. GET /api/v1/uns/broker/status (topic read permission) reports running, clients, maxClients and clientIdInUseRefusals of the replica that answered. It has no cluster block. The cluster block (broker.cluster: replica, mesh, members, memberCount, linksUp, and, only when set, storeUnreachable, fenced, duplicateReplicaId, belowFormatFloor, sessionLogValuesMissed, sessionLogReadsHeld) is in GET /api/v1/system/uns/settings/status, which needs uns_settings:update. The MQTT Broker page reads GET /api/v1/system/uns/broker/status (same permission), which adds each replica's pod name, devices and saved sessions, the replicas that left, and which replica answered (answeredBy). GET /api/v1/uns/health needs no sign-in and says only healthy or not, never why. A request through the gateway reaches any replica, so read a specific replica through its pod as above.

Who to escalate to. When an entry says escalate, open a case with MaestroHub support and attach: the alert and its start time, the chart version and image tags of the UNS pods (kubectl -n <app-ns> get pods -l app.kubernetes.io/name=uns -o wide), the UNS logs of every replica from 15 minutes before the alert, and the history of the alert's metric.

Ingest alerts​

MaestroHubUNSIngestStoreUnavailable​

Severity: warning. Fires when maestrohub_uns_broker_ingest_store_retries_total{outcome=~"redelivery|hung"} rose in the last 10 minutes, for 5 minutes.

What it means. The historian did not answer MaestroHub's writes. A device's QoS 1 or 2 value is acknowledged only once it is stored or refused, so while the historian does not answer, values wait and are retried, backing off to every 2 seconds, for up to 30 seconds. Two outcomes trip the alert:

  • redelivery: a value's store stayed unavailable for the whole 30 seconds. It is refused as internal_error: an MQTT 5 device gets PUBACK 0x80; an MQTT 3.1.1 device is disconnected without its PUBACK and sends it again when it reconnects.
  • hung: one store attempt ran past 25 seconds, ignoring its deadline. The device's connection is ended without an answer, so it sends that value and the ones behind it again.

How bad. Values are not stored while it lasts. An MQTT 5 client library does not resend a refused value by itself, so a device that does not act on the refusal loses it. Live subscribers see nothing new for the affected topics.

Confirm.

  • sum by (pod, outcome) (increase(maestrohub_uns_broker_ingest_store_retries_total[10m])). retried alone is a failover that recovered; redelivery and hung are what the alert counts.
  • The UNS log, one line per outage: embedded broker: values could not be stored — their publishes stay unacknowledged and are retried; ..., then per value embedded broker: the store stayed unavailable for a value's whole retry budget — the value is refused ... or embedded broker: a store attempt has run past its limit, ignoring its deadline — .... Recovery: embedded broker: values are being stored again — the held publishes are acknowledged as they land.
  • The historian's state: Context Engine > Run > Historian, or storage in GET /api/v1/system/uns/settings/status (healthy, message).

Likely causes. TimescaleDB down, failing over or out of connections; a long lock on the signal tables; a full disk under the historian; on Pebble, a disk that stalls.

Fix.

  1. Open the Historian page. If it shows Down, fix the database first: is it reachable from the UNS pods, is it accepting connections, is its disk full?
  2. On TimescaleDB, look for blocked writers and long transactions, and for max_connections exhausted (each UNS replica opens up to Max Open Connections).
  3. Watch the log for values are being stored again. Held values are then acknowledged as they land.
  4. Tell the owners of MQTT 5 devices that values refused during the outage are in the reject feed (Alerts › Ingress rejects) and are not stored unless the device sends them again.

Escalate when the historian is healthy and the alert still fires, or when hung keeps rising after the database looks idle.

MaestroHubUNSIngestQuarantined​

Severity: warning, as on the Overview card: one value whose fate is unknown, while every other value keeps being stored. Fires on the first evaluation where maestrohub_uns_broker_ingest_store_retries_total{outcome="quarantined"} rose in the last 15 minutes.

What it means. One value's write hung (ran past 25 seconds) on 5 deliveries within an hour. Ending the connection again would only start another round, so the broker gives up on it: an MQTT 5 device gets PUBACK 0x80, and an MQTT 3.1.1 device, which has no reason codes, is acknowledged, so the device stops resending it. The hung attempts keep running, so the value may still end up stored.

How bad. Whether that value is stored is not known. Each quarantined value is one the device will not send again.

Confirm.

  • sum by (pod) (increase(maestrohub_uns_broker_ingest_store_retries_total{outcome="quarantined"}[15m])).
  • The UNS log names the device and the topic: embedded broker: a value's write hung past its limit on every delivery — it is quarantined: ..., with client_id and topic.
  • The same value's earlier rounds were counted as outcome="hung".

Likely causes. A write the historian holds on every attempt: a lock on that series, a row the database cannot take quickly, a disk that stalls on one file.

Fix.

  1. Take client_id and topic from the log line.
  2. Read the topic's history around the time of the log line to see whether the value landed.
  3. On TimescaleDB, look for locks on the topic's series and for long transactions. On Pebble, check the disk's latency.
  4. If the value is missing and it matters, have the device or its owner publish it again once the historian answers normally.

Escalate always when the cause is not a lock or a disk you can find: a write that hangs on the same value every time is worth a support case, with the log lines of all its rounds.

MaestroHubUNSLiveMirrorDropping​

Severity: warning. Fires when maestrohub_uns_broker_live_mirror_dropped_total rose in the last 10 minutes, for 10 minutes, by reason.

What it means. Every stored value is published again on its canonical topic so live subscribers (dashboards, apps, other systems) see it. That publish failed. The value is stored; only its live copy is missing.

How bad. Live subscribers miss values. History is complete.

Confirm.

  • sum by (pod, reason) (increase(maestrohub_uns_broker_live_mirror_dropped_total[10m])).
  • reason="not_connected": the broker is not running on that replica. running in GET /api/v1/uns/broker/status is false, or the MQTT Broker page shows Down.
  • reason="error": the broker refused the publish. Read the UNS log of that pod around the same minutes.

Likely causes. A broker restart after a settings change (short); a broker that failed to start; on a clustered replica, a session log or retained store that did not take the value in time (see MaestroHubUNSBrokerSessionLogAppendsRefused).

Fix.

  1. If the broker is down on that replica, read its log for why it did not start, and fix that first.
  2. If it is up and the reason is error, check whether the session-log alerts fire too, and follow those.
  3. Live subscribers recover on their own once the publishes succeed. Values they missed are in history.

Escalate when the broker is running, no session-log alert fires, and reason="error" keeps rising.

Broker cluster alerts​

These read series only a clustered broker reports. A single-node installation never fires them.

Severity: warning. Fires when a replica's maestrohub_uns_broker_cluster_mesh_links_up is below its maestrohub_uns_broker_cluster_members minus 1, for 5 minutes.

What it means. The replicas pass live values to each other over a mutually authenticated TLS mesh on port 7884. This replica sees more replicas than it has links to. A live value published on it does not reach the MQTT subscribers and in-process handlers of the replicas it is not linked to.

How bad. Live subscribers on the other replicas miss values published here. Stored values are not affected, and persistent sessions are not either: they read their session log in NATS, not the mesh.

Confirm.

  • max by (pod) (maestrohub_uns_broker_cluster_mesh_links_up) against max by (pod) (maestrohub_uns_broker_cluster_members).
  • The Cluster row on the MQTT Broker page names the replicas this one is not linked to, for example 3 replicas, not linked to maestrohub-uns-2. Through the API: broker.cluster.members[].linked in GET /api/v1/system/uns/settings/status.
  • The UNS log: mesh: link to another replica down; reconnecting, and possibly mesh: refused a connection without a valid handshake (certificate), mesh: refused the link to another replica: no mesh protocol version in common (two releases too far apart), or mesh: still refusing connections from a replica that is not among this one's members.

Likely causes.

  • A NetworkPolicy that blocks 7884 between the UNS pods (the chart's allow-uns-mesh policy admits it when networkPolicy.enabled).
  • A replica with another mesh certificate. The chart keeps one certificate per deployment in the Secret <release>-uns-mesh-tls; after it is rotated, an old and a new replica refuse each other until every replica has rolled.
  • Two releases with no mesh version in common (more than one release apart).
  • A replica that has just started: it links within seconds, which the 5 minutes leave alone.

Fix.

  1. Check the policies: kubectl -n <app-ns> get networkpolicy. If you run your own, admit TCP 7884 between pods labelled app.kubernetes.io/name: uns.
  2. If the log says a handshake was refused, restart the UNS StatefulSet so every replica carries the current <release>-uns-mesh-tls: kubectl -n <app-ns> rollout restart statefulset/<release>-uns.
  3. If the log says no version in common, finish the rolling upgrade so every replica runs one release.

Escalate when the port is open, every replica runs the same image and certificate, and links still do not come up.

MaestroHubUNSBrokerSessionLogValuesMissed​

Severity: warning. Fires on the first evaluation where maestrohub_uns_broker_session_log_values_missed_total rose in the last 15 minutes. One missed value is enough.

What it means. The QoS 1 and QoS 2 values a workspace's persistent sessions want are kept in one NATS JetStream log per workspace, for maxSessionExpiry and up to brokerSessionLogMaxBytes (256 MiB by default). A session that was away, or behind, longer than that found the oldest values gone before it read them. Its device will not receive them.

How bad. Each counted value is one a device never got over MQTT. The values are stored: read them from history.

Confirm.

  • sum by (pod, reason) (increase(maestrohub_uns_broker_session_log_values_missed_total[15m])). The only reason is evicted.
  • The UNS log, once per connection: broker cluster: a persistent session missed values: its session log dropped them (size cap or age) before it read them, with client_id, workspace, missed, from_seq and to_seq.
  • The mqtt-broker component of that pod's GET /health ends with persistent sessions missed N values their session log dropped before they read them (raise brokerSessionLogMaxBytes).

Likely causes. A device away for longer than its workspace's log lasts at the rate the workspace's sessions want values (a wide subscription such as # fills it fastest), or a session reading more slowly than values arrive.

Fix. Follow When values were missed.

Escalate when the log lasts far longer than the session was away (for example a few seconds away and values missed).

MaestroHubUNSBrokerSessionLogAppendsHeld​

Severity: warning. Fires when maestrohub_uns_broker_session_log_appends_held_total rose in the last 10 minutes, for 10 minutes.

What it means. Values persistent sessions want are appended to their workspace's log in NATS JetStream before the device's acknowledgment. JetStream takes them more slowly than devices send them, so values wait for room, their devices' acknowledgments with them, for up to brokerSessionLogAppendWait (10 s by default). The same counter rises while a workspace's log is not open yet, for example while NATS is down.

How bad. Nothing is lost yet. Devices see slow acknowledgments. Past the wait, publishes are refused (next alert).

Confirm.

  • sum by (pod) (increase(maestrohub_uns_broker_session_log_appends_held_total[10m])).
  • The UNS log: broker cluster: values wait for room to be appended to their session log, their devices' acknowledgments with them; ... or broker cluster: values wait for their session log to open, or for this replica's view of the persistent sessions to catch up, ....

Likely causes. NATS JetStream disk slow or near full; NATS servers short of CPU; a log whose stream leader moved or is electing; a burst far above the usual rate.

Fix.

  1. Check the NATS servers: pod restarts, CPU, and disk use against nats.config.jetstream.fileStore.maxSize.
  2. Check the workspace's log: nats stream info LOG_mqtt-sessions-<workspace id> (see Connecting the nats CLI). Look at its leader and its replicas' lag.
  3. If NATS is healthy and the rate is simply higher than it takes, raise brokerSessionLogAppendWait (Helm uns.mqtt.embedded.sessionLogAppendWait) as a stopgap and size NATS for the rate.

Escalate when NATS shows no pressure and the counter keeps rising.

MaestroHubUNSBrokerSessionLogAppendsRefused​

Severity: warning. Fires on the first evaluation where maestrohub_uns_broker_session_log_appends_refused_total rose in the last 10 minutes.

What it means. After waiting brokerSessionLogAppendWait, a value still found no room in its session log (or the log stayed closed, or NATS refused its appends for 15 seconds). The publish was refused: an MQTT 5 device gets PUBACK 0x80, an MQTT 3.1.1 device's connection is ended after the acknowledgments before it. Nothing was acknowledged and then lost; the device sends it again. A retained value the cluster's retained store did not take within the same wait is refused the same way.

How bad. Devices are being refused. An MQTT 5 device that does not act on PUBACK 0x80 loses the value.

Confirm.

  • sum by (pod) (increase(maestrohub_uns_broker_session_log_appends_refused_total[10m])).
  • The UNS log: broker cluster: publishes refused: their values found no room in their session log in time; ... or broker cluster: publishes refused: their session log did not take their values in time (not open, or refusing appends); ....
  • Usually MaestroHubUNSBrokerSessionLogAppendsHeld fired first.

Fix. As for the held appends above: NATS first. If NATS was down, the counter stops once it is back and the logs open again.

Escalate when NATS is healthy and refusals continue.

MaestroHubUNSBrokerSessionLogAppendsReordered​

Severity: warning. Fires on the first evaluation where maestrohub_uns_broker_session_log_appends_reordered_total rose in the last 10 minutes.

What it means. A value was stored in its workspace's session log ahead of an earlier value of the same topic. Either the earlier value's append was refused and is being retried, or the log stored the two out of order. Persistent sessions reading that topic get the two values out of order.

How bad. Nothing is lost: both values are in the log and in history. A subscriber that keeps only the last value it received can hold the older one until the next value arrives.

Confirm.

  • sum by (pod) (increase(maestrohub_uns_broker_session_log_appends_reordered_total[10m])).
  • The UNS log, at most once a minute: broker cluster: a value was stored in a session log ahead of an earlier value of its topic ..., with the workspace and the last topic.
  • A NATS JetStream stream leader change at the same time: nats stream info LOG_mqtt-sessions-<workspace id> (see Connecting the nats CLI).

Likely causes. A leader change of the workspace's log stream (a NATS server restarted or was rescheduled). Retried appends after MaestroHubUNSBrokerSessionLogAppendsHeld or refusals.

Fix. Usually none: it settles once the stream has a stable leader. If the counter keeps rising, find why the leader keeps changing: NATS pod restarts, its disk, its CPU, the network between the NATS servers.

Escalate when NATS is stable and the counter keeps rising.

MaestroHubUNSBrokerSessionSubscriptionsKept​

Severity: critical. Fires on the first evaluation where maestrohub_uns_broker_session_subscriptions_kept_total rose in the last 10 minutes. One is enough.

What it means. Every replica stores each persistent session's subscriptions in NATS. They change only when the device sends a SUBSCRIBE or an UNSUBSCRIBE, or starts a new session. A write of the session's state tried to change them otherwise: from a connection that never loaded the session (for example one refused while it connected), or from a write that only marks the session's owner or its disconnect. The broker refused that part of the write and kept the stored subscriptions.

How bad. Nothing is lost yet: the device keeps its subscriptions and keeps receiving. But a code path in MaestroHub tried to shrink what a device subscribed to. Had the write gone through, every later reconnect, on any replica, would have found the session present and delivered nothing.

Confirm.

  • sum by (pod) (increase(maestrohub_uns_broker_session_subscriptions_kept_total[10m])).
  • The UNS log, at ERROR, one line per write: broker cluster: a persistent session's stored subscriptions were kept against a write that would have changed them without a SUBSCRIBE or UNSUBSCRIBE, with client_id, why, and how many subscriptions were stored and written.

Fix. Nothing to change on the deployment: the broker already protected the session. Check that the named device still receives (the MQTT Broker page, or a test subscription).

Escalate always: report the log lines (client ID, why, counts) and what happened around them (a fence, a NATS outage, a rolling restart).

MaestroHubUNSBrokerRoutesLate​

Severity: warning. Fires when maestrohub_uns_broker_routes_late_total rose in the last 10 minutes, for 5 minutes.

What it means. Before a SUBACK, the broker stores the subscription's route so the other replicas know to forward matching values here. That write took longer than 2 seconds or failed, so the SUBACK went first. Values published on this replica reach the subscriber. Values published on other replicas miss it until the route is stored; the broker keeps retrying it.

How bad. New subscribers miss values published elsewhere for a while. Existing subscriptions are not affected.

Confirm.

  • sum by (pod) (increase(maestrohub_uns_broker_routes_late_total[10m])).
  • The UNS log: broker cluster: a subscription's route could not be published before its SUBACK; the sync loop retries — until then, values published on other replicas do not reach it.

Likely causes. NATS slow or unreachable from this pod; the mqtt-routes bucket's stream leader electing.

Fix. Check NATS (as for the session log alerts) and the network between this pod and NATS. If the replica cannot reach NATS at all it fences itself within seconds; see When a replica is fenced.

Escalate when NATS is healthy and routes stay late.

MaestroHubUNSBrokerMeshValuesLost​

Severity: warning. Fires on the first evaluation where maestrohub_uns_broker_cluster_mesh_values_lost_total rose in the last 10 minutes.

What it means. Each replica holds at most 64 MiB of live values for each other replica, waiting for that replica to acknowledge them. While a link was down, or this replica read more slowly than another sent, that buffer overflowed and the oldest values were dropped.

How bad. Live subscribers on this replica missed values. Persistent sessions did not: they read their session log. The values are stored.

Confirm.

  • sum by (pod) (increase(maestrohub_uns_broker_cluster_mesh_values_lost_total[10m])) on the receiving replica.
  • On the sending replica: mesh: values for another replica were dropped before they were sent: its link is down or it reads slower than this replica sends. On the receiving one: mesh: values from another replica were lost.

Likely causes. A mesh link down for a while (see MaestroHubUNSBrokerMeshLinksDown); this replica short of CPU; network congestion between the UNS pods.

Fix.

  1. If links are down, fix those first.
  2. Check this pod's CPU throttling and its limits; raise them if it is throttled.
  3. Check the network between the UNS pods.

Escalate when links are up, CPU is free, and values are still lost.

MaestroHubUNSBrokerClusterFormatUnreadable​

Severity: critical. Fires on the first evaluation where maestrohub_uns_broker_cluster_format_unreadable_total rose in the last 10 minutes, by kind (message or cursor).

What it means. A replica met cluster state written in a format newer than it reads. The version policy keeps every replica able to read what the others write, so this means a replica of an older release runs beside newer ones that already write a newer format.

How bad. A persistent session whose log entry this replica cannot read is held there: its later values wait behind it. None is lost: once a replica that reads the format serves the session, it gets them all, in order.

Confirm.

  • sum by (pod, kind) (increase(maestrohub_uns_broker_cluster_format_unreadable_total[10m])).
  • The UNS log: broker cluster: a persistent session's log reader holds at a value written in a newer format than this replica reads; its later values wait behind it, with client_id and seq.
  • The mqtt-broker component of that pod's GET /health ends with N persistent sessions wait at a value of a newer format than this replica reads (finish the rolling upgrade).
  • The image of each UNS pod: kubectl -n <app-ns> get pods -l app.kubernetes.io/name=uns -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}{end}'.

Fix. Follow When a session log is held at an unreadable format.

Escalate when every replica runs the same image and the counter still rises.

What to do when​

No broker replica answers​

What happens. Every UNS replica needs NATS: for its membership in the cluster and for the stores it shares with the others. When NATS is unreachable (every NATS server down, or the network between the UNS pods and NATS cut), every replica fences itself within about 12 seconds: it closes its MQTT connections and refuses new ones. A replica that cannot reach the shared stores for 15 seconds also reports itself not ready. A replica that is fenced or not ready answers 503 on /health/ready, so Kubernetes takes it out of its Services: out of the MQTT Service <release>-uns-mqtt, so devices cannot connect, and out of the Service the API reaches it through, so the page's status read gets no answer either. The pods are not restarted; each serves again on its own once NATS answers.

The same verdict appears whenever no UNS replica is ready for another reason: every UNS pod crashing, pending or scaled to 0.

How it shows.

  • The MQTT Broker page: No replica answered, with No UNS replica answered. When NATS is unreachable every replica steps aside and leaves the Service, so devices cannot connect either. Check NATS first, then the UNS pods. The last answer stays below, dimmed, with the time it was read.

  • No UNS pod is ready, and the MQTT Service has no endpoints:

    kubectl -n <app-ns> get pods -l app.kubernetes.io/name=uns
    kubectl -n <app-ns> get endpoints <release>-uns-mqtt

  • Readiness of one pod (the port-forward under Before you start): 503 with broker-cluster-store: the broker cannot reach its cluster stores (since ...), or the fenced reason (see A replica is fenced).

  • Every replica logs broker cluster: this replica fences itself: ....

  • MaestroHubPodNotReady fires if it lasts 15 minutes.

What to do.

  1. Check NATS first. kubectl -n <deps-ns> get pods -l app.kubernetes.io/name=nats, and the NATS logs. Bring NATS back: no replica serves MQTT until it answers.
  2. Check the way from the UNS pods to NATS when the NATS pods are healthy: NetworkPolicy, DNS, the node network.
  3. Then the UNS pods. When NATS answers and the pods are still not ready, read the readiness reason of each, and kubectl -n <app-ns> describe pod <pod> for a pod that is pending or crashing (kubectl -n <app-ns> logs <pod> --previous for the last crash).
  4. Recovery is on its own. Once NATS answers, each replica registers again, its readiness passes, and Kubernetes sends it connections again. Devices reconnect and take up their saved sessions from NATS JetStream, within their session expiry (see Saved sessions during a whole-cluster outage). Nothing the broker acknowledged is lost; values devices could not send meanwhile are sent when they reconnect, if the device keeps them.

On a single machine the page has no replica to lose: No replica answered there means the MaestroHub process did not answer. Check that it runs (docker compose ps, the container's log, or the service of the native binary).

A replica stepped aside​

What happens. A replica left the cluster: its membership entry expired, the others no longer count it as a member, and no new member runs under its pod name. While it is out it takes no MQTT connections. Its devices reconnect to the other replicas behind the same Service, and each persistent session is taken up there from NATS JetStream, with its subscriptions and the values kept for it.

A pod that restarted is not this. Every start of a pod is a new member under the same pod name, so the page shows the new member as Serving with Restarted 12 s ago: nothing to do.

How it shows.

  • The top line: Degraded: 2 of 3 replicas serving (or Degraded: 2 replicas serving without monitoring.metrics:read), naming the replica, for example maestrohub-uns-2 stepped aside: not a member for 41 s. It adds No acknowledged value was lost. when the loss counters of the last 24 hours are known and zero.
  • Its tile: Stepped aside, Not a member for 41 s, Takes no connections until it rejoins, and how many of its devices are reconnecting.
  • Clients: the devices still recorded on it are listed last, Reconnecting: was on maestrohub-uns-2, and are not counted as connected. They move up as they reconnect to another replica.
  • Once the cluster has taken the replica for gone (its devices' wills and sessions handed over), its tile goes. Someone who holds monitoring.metrics:read then sees its pod as Not reporting while it is still not a member; without that permission the page no longer counts it, and the top line can read Healthy with one replica fewer.

What to check.

  1. Find the pod named on the tile: kubectl -n <app-ns> get pods -l app.kubernetes.io/name=uns -o wide.
  2. Running but not ready: read its readiness reason (the port-forward under Before you start). Fenced: A replica is fenced. Cannot reach the cluster stores: its way to NATS (NetworkPolicy, node, DNS); when it is every replica, see No broker replica answers. Below the format floor: A replica is below the format floor. Another process under its replica ID: an old pod that was not fully stopped.
  3. Pending: kubectl -n <app-ns> describe pod <pod> says why it is not scheduled (node lost or drained, resources, volume).
  4. Crashing: kubectl -n <app-ns> logs <pod> --previous.
  5. Devices that stay in Reconnecting do not come back on their own: check the device's reconnect setting and that it reaches the Service address shown under Connect a device.
  6. Values. If the top line does not say No acknowledged value was lost., look at Values a saved session missed under Your data, and at Values were missed. What devices published is in the historian either way.

When the pod is ready again it joins as a new member and its tile is Serving. The other replicas carry its devices meanwhile, up to their maxClients each: the Connection limits row under Security shows how close they are.

Devices cannot connect after a port change​

What happens. A device reaches the broker through a chain of ports: the port it is configured with, the port the deployment exposes (a Kubernetes Service, a NodePort or load balancer, a port Docker publishes), and the port the broker listens on. Where the deployment owns the listeners (deploymentOwns includes listeners), the broker listens on the deployment's ports, read at every start; a port changed in one place but not the others breaks the chain.

How it shows. Devices are refused or time out at connect, and the Clients tab shows fewer connected than expected. Under Connect a device, each listener tile shows the port the running broker serves, with (devices: N) when a public port is set, and the address shows the port devices should use.

What to do, by deployment.

  1. Kubernetes. The listener, the Service, the NetworkPolicy and the pod ports all come from the same Helm values (uns.mqtt.embedded.tcpPort, uns.mqtt.embedded.tlsPort, uns.mqtt.embedded.webSocket.port), so after a rollout they agree. Check what the Service exposes: kubectl -n <app-ns> get svc <release>-uns-mqtt. For a NodePort or a load balancer, the port devices use is the Service's outer port, not the listener's: point the devices at it, and set uns.mqtt.service.publicPorts (for example {mqtts: 30883}) so the page shows it. A port saved on the page cannot apply here: the save is refused with 409 BROKER_SETTING_MANAGED_BY_DEPLOYMENT.
  2. Docker Compose with deploymentOwns: [mode, listeners]. The broker listens on the ports in config.yaml (modules.uns.defaults.broker.embedded.tcpPort, .webSocketPort), inside the container; docker-compose.yaml publishes host ports to them ("${MQTT_PORT:-1883}:1883"). The container side of the mapping must equal the port in config.yaml; then docker compose up -d to apply. A changed host port (MQTT_PORT) changes only what devices use: set it as the public port under Settings, Public address, so the page shows it. A port that docker-compose.yaml does not publish is not reachable from outside the host.
  3. Docker Compose without deploymentOwns (a config.yaml from before it existed): a port saved on the page moves the listener away from the port the compose file publishes, and devices can no longer reach it. Add deploymentOwns: [mode, listeners] under modules.uns.defaults.broker in config.yaml and restart: the ports in config.yaml apply again, and the page shows them read-only. Or set the port on the page back to the published one.
  4. The native binary. The page owns the ports: after a save the broker listens on the new port at once. Open it in the host's firewall and change it on every device.

Saving a listener port on a single machine restarts the broker, so every device reconnects, and the saved sessions this machine keeps in memory for devices that are away end (see Saving a broker change).

A replica is fenced​

What happens. Each replica keeps a membership entry in NATS (the mqtt-members bucket). The entry expires 10 seconds after its last write, and the replica rewrites it every second (also while earlier writes are still on their way, and a write that lands late counts from when it was sent), so a NATS slow but answering fences nobody up to about 6 seconds per round trip, and fences the replicas repeatedly from about 8 seconds (Supported NATS latency; the timing scales with brokerMemberTTL: Membership timing). When the entry has been missing for 5 more seconds, on their own clock, the other replicas take the replica for gone: they send its devices' wills (after the grace below) and take over their sessions. A replica cut off from NATS still holds its MQTT connections, so before that can happen it fences itself: about 11.7 seconds after the last write that landed (on its monotonic or its wall clock, whichever says longer), or at once when it finds its entry gone from the store, it closes every MQTT connection (MQTT 5 clients get DISCONNECT 0x8B, server shutting down), hands its devices' wills over, and refuses every new CONNECT (server unavailable) until a write of its entry lands again. Its devices reconnect to the other replicas.

How it shows.

  • Log, at ERROR: broker cluster: this replica fences itself: it closes every MQTT connection and accepts none until it is a registered member again, so the other replicas can decide for its clients, with the reason and fence_generation, then broker cluster: fenced: closed every MQTT connection of this replica; its clients reconnect to the other replicas. When it is a member again: broker cluster: this replica is a registered member again; it accepts MQTT connections again, with the next fence_generation.
  • If the others took it for gone (its entry stayed missing past the 5 s dwell), the replica that acted on it (the elected sweeper) logs once at INFO broker cluster: a replica was taken for gone: …, with gone_replica, absent_for (how long its entry had been missing), and what was done: sessions_marked_disconnected, wills_handed_over, client_ids_released, routes_removed. It also counts it: maestrohub_uns_broker_cluster_members_gone_total (no labels; once per gone replica, on the replica that acted). A fence the others never acted on leaves it flat. A replica that rejoins after being taken for gone is a new event if it is taken for gone again. sum(increase(maestrohub_uns_broker_cluster_members_gone_total[1h])) shows how often replicas were taken for gone; a replica restarting is one each time (its id is new per start).
  • Refused CONNECTs: maestrohub_uns_broker_connect_refused_total{reason="fenced"}, and at most once a minute broker cluster: CONNECTs refused: this replica is fenced (not a registered member of the cluster).
  • Readiness: GET /health/ready answers 503 with a reason such as broker-cluster-store: the broker replica is fenced: no write of its member entry landed for 11.666666667s (the entry expires 10s after the last one, and the others take it for gone 5s after that): the other replicas may take it for gone, or ... its member entry is gone from the store (deleted, or expired early): the other replicas may have taken it for gone. Kubernetes takes the pod out of the MQTT Service. The pod is not restarted.
  • Broker status: broker.cluster.fenced carries the same reason, and the broker is reported unhealthy with the broker replica is fenced: … as its message.
  • If NATS stays unreachable for 15 seconds, the reason becomes broker-cluster-store: the broker cannot reach its cluster stores (since ...), and storeUnreachable appears in the cluster status.
  • MaestroHubPodNotReady fires if it lasts 15 minutes.

How it recovers. On its own. As soon as a write of its entry lands (or, if the entry expired, the replica registers again), it logs broker cluster: this replica is a registered member again; it accepts MQTT connections again and readiness passes. After a longer store outage: broker cluster: the shared stores are reachable again; this replica is ready again.

What to do.

  1. Find what cut it off: the NATS pods (kubectl -n <deps-ns> get pods -l app.kubernetes.io/name=nats), and the network between this pod and NATS (NetworkPolicy, node, DNS).
  2. If every replica is fenced at once, NATS itself is down or unreachable: no replica serves MQTT until it is back. Fix NATS.
  3. If one replica stays fenced with NATS healthy, restart that pod.

NATS refusing one replica's guarded membership writes. After the NATS servers' routes were cut and healed, the new leader of the KV_mqtt-members stream was seen refusing every compare-and-swap write of one replica's entry ("wrong last sequence") for as long as it stayed leader, while it held the entry unchanged and took every other write of it. The replica keeps its membership through it: it writes the entry without the revision check, as it does while NATS is slow, and stays a member (before, it fenced and registered again every 10 s for as long as it lasted). It says so once, at WARN: broker cluster: the store refuses compare-and-swap writes of this replica's member entry although it holds the entry at that revision; the heartbeat writes it unguarded, as a pipelined beat, with member and revision. Nothing is lost meanwhile, but each heartbeat costs three NATS round trips instead of one. A change of the stream's leader ends it: nats stream cluster step-down KV_mqtt-members.

Related readiness reasons. another process registered under this broker replica's ID (<id>): two processes run under one replica ID; on Kubernetes the ID is the pod name, so this points to a pod that was not fully stopped. And the format floor, next.

A replica is below the format floor​

What happens. When every live replica reads a newer format, the replicas first record it as the cluster's floor in the mqtt-cluster-floor bucket, then switch to it. The floor only goes up. A replica that reads less than the floor must not join, because it would misread what the cluster stores.

How it shows.

  • At start, the replica refuses to start. Log, at ERROR: broker cluster: this replica is too old for the formats the cluster writes and refuses to start, with format, needed, reads, floor_raised_by and remediation. The UNS service on that pod fails its start with this error:

    broker cluster: the cluster needs <format> format <needed> and this replica reads up to <reads> (the floor was raised by replica "<id>" at <time>): upgrade this replica to the release that replica runs, or a later one — the floor never goes down by itself, so a rollback past it is not possible

  • While running, if the floor rises above what it reads: broker cluster: the cluster's format floor rose above what this replica reads; it is not ready. The broker reports itself unhealthy with the same error text, readiness answers 503 with it, and the replica fences itself and stays fenced.

What to do.

  1. Read floor_raised_by in the log: the release that replica runs reads the floor.
  2. Run this replica on that release, or a later one. On Kubernetes: make sure the StatefulSet's image is the newer release, then delete the old pod so it starts on it.
  3. Do not try to lower or delete the floor. The cluster already stores data in the newer format, and an older replica would misread it.

A rollback has a limit for this reason: roll back before the last replica of the old release is replaced. See Rolling upgrades.

A session log is held at an unreadable format​

What happens. A replica of an older release serves a persistent session whose log holds a value in a format that release does not read. The replica never skips it: the session's reader holds there, its later values wait, and none is lost. This is what MaestroHubUNSBrokerClusterFormatUnreadable reports.

What to do.

  1. List the release of every UNS pod (command in the alert entry).
  2. Finish the rolling upgrade: every UNS replica on the same release. kubectl -n <app-ns> rollout status statefulset/<release>-uns shows whether a rollout is stuck.
  3. Once the session is served by a replica that reads the format (its replica upgraded, or the device reconnects to an upgraded one), it gets the held value and everything after it, in order.
  4. If you must go back instead, roll back the newer replicas only while the floor has not been raised (see the previous section).

Values were missed​

Counted in maestrohub_uns_broker_session_log_values_missed_total; alert MaestroHubUNSBrokerSessionLogValuesMissed.

  1. Find the sessions. Each connection that missed values logs broker cluster: a persistent session missed values: ... once, with client_id, workspace, missed, from_seq and to_seq (further misses on the same connection log at DEBUG).
  2. Recover the data. The values are stored. Read the time window from history for the topics the device subscribes to.
  3. Size the log. Each workspace's log keeps values for maxSessionExpiry and up to brokerSessionLogMaxBytes, oldest out first. Size it for what the workspace's persistent sessions want over the longest time one stays away. Every start logs the cap and how long it lasts at an example rate: broker cluster: persistent sessions' values are kept per workspace up to this size and age; ..., with lasts_at_1000_values_per_second_of_1KB.
  4. Raise the cap. Helm uns.mqtt.embedded.sessionLogMaxBytes (default 268435456); config.yaml modules.uns.brokerSessionLogMaxBytes (0 means 256 MiB, at least 1 MiB). JetStream reserves the cap for every workspace that has persistent sessions, on every NATS server, so raise nats.config.jetstream.fileStore.maxSize (and the volume, nats.config.jetstream.fileStore.pvc.size) with it.
  5. Narrow subscriptions. A persistent session subscribed to # makes its workspace's log keep every QoS 1 and 2 value. Subscribe to what the device needs.

Client ID conflicts​

Client IDs are broker-wide, not per workspace. A CONNECT whose client ID is held (connected, or kept for a disconnected persistent session) by another identity, in any workspace, is refused.

  • What the device sees. MQTT 5: CONNACK client identifier not valid (0x85) with a reason string. MQTT 3.1.1: identifier rejected.
  • Counted in maestrohub_uns_broker_connect_refused_total{reason="client_id_in_use"}, and in clientIdInUseRefusals of GET /api/v1/uns/broker/status (since the answering replica started). The MQTT Broker page shows Client ID conflicts once there is one.
  • Logged at most once a minute per user name (per source address when trustClientAddresses is on): authz-hook: CONNECT refused — the client id is held by another identity (client ids are broker-wide, not per workspace), with client_id, workspace, held_by_workspace and, when known, held_by_workspace_slug.

What to do.

  1. Take client_id and held_by_workspace from the log.
  2. Give every device its own client ID. Two devices sharing one, even in different workspaces, take turns being refused.
  3. A session kept for a disconnected client releases its client ID when it expires (maxSessionExpiry) or when its own client reconnects with a clean start.

The same identity connecting with its own client ID on another replica is not a conflict: the newer connection takes the session over.

A device's will after a restart​

A replica going away is not its devices going offline. When a replica stops (a rolling restart), fences itself, or is found gone, the Last Will of each device connected to it waits for that device to reconnect: for brokerWillStopGrace (30 s by default), or the will's own Will Delay Interval if that is longer. A device that reconnects to any replica in that time does not have its will sent, whatever its Clean Start. One that does not is sent its will once, by another replica, after its credential is checked again. A will whose owner's credential was revoked is not sent (maestrohub_uns_broker_wills_withheld_total, and broker cluster: a will was withheld in the log).

This hand-over works across replicas once the cluster writes session-state format 2, that is, once every replica runs a release that keeps wills in the cluster (the mqtt-session-will bucket). During a rolling upgrade from the release before, each replica keeps its own connections' wills, as that release did.

How it shows. A stopping replica logs how many wills it handed over: broker cluster: stopping: the wills of this replica's connections are left to the other replicas, sent only if their devices do not reconnect within the grace (a stop is not the devices going offline), with wills_handed_over and grace.

What to do.

  • Wills arrive after a restart although devices came back: the devices took longer than the grace to reconnect. Raise brokerWillStopGrace (Helm uns.mqtt.embedded.willStopGrace, a Go duration such as "60s") above your devices' reconnect time.
  • Wills arrive late: that is the grace at work. Lower it if consumers need wills sooner, accepting that a rolling restart then sends wills for devices that are reconnecting.
  • A single-node broker (no cluster) that stops or restarts sends no undelayed wills at all.

The NATS logins switch​

global.natsAuth signs every service in to NATS as its own users. The broker's cluster state then lives in its own account, BROKER, which only the user uns-broker joins; no other service can read or write it. values-production.yaml turns it on.

Turning it on (or off) for a NATS cluster that already runs is not a rolling upgrade: NATS has to be stopped and its JetStream volumes removed first, and the chart refuses the upgrade until then. For the broker this means everything in JetStream is lost: persistent MQTT sessions, the values kept for them, retained messages, wills and the format floor. Devices reconnect and subscribe again on their own; a retained value comes back when its device publishes it again. Databases, including UNS history, are untouched.

Follow the steps in the chart's UPGRADING.md, section NATS authentication (global.natsAuth). Plan it for a maintenance window and tell device owners that persistent sessions start empty.