How the broker cluster behaves
What survives a failure, what each QoS level guarantees, and the limits that bound the broker.
- Applies to
- Kubernetes chart, every UNS replica serves MQTT
- Shared state
- NATS JetStream
- Live values between replicas
- mTLS mesh on port 7884
In an Enterprise deployment every UNS replica runs the built-in MQTT broker, and the replicas act as one broker behind the Service <release>-uns-mqtt. This page is for operators: what to expect when something fails. The settings themselves are on MQTT broker and historian; what to do when an alert fires is on MQTT broker runbooks.
The parts
- The replicas. Pods
<release>-uns-0,-1, … of the StatefulSet<release>-uns. Any replica serves any client. Each start of a pod joins the cluster as a new member: its member ID is the pod name with a suffix that is new at every start, so a restarted pod is never mistaken for the process it replaced. - NATS JetStream holds what the replicas share: who is a member, which replica wants which topics, who owns each client ID, every persistent session's state and read position, the values kept for persistent sessions (one log per workspace), QoS 2 flows, wills, retained messages and the format floor. The full list is on Back up and restore the broker's NATS state. Each stream keeps
global.natsReplicascopies (3 by default). - The mesh carries live values from the replica a value was published on to the replicas whose subscribers want it, over mutually authenticated TLS on port
7884. - Membership. Each replica rewrites its member entry every second; the entry expires 10 seconds after its last write, and the others take a replica for gone only once its entry has been missing for 5 more seconds on their own clock. A replica none of whose writes landed for about 11.7 seconds, counted from when that write was sent, fences itself (closes its MQTT connections and refuses new ones) before the others can take it for gone; one that finds its entry gone from the store fences at once, and registers again once its connections are closed. See When a replica is fenced.
- A slow NATS fences nobody, up to about 6 seconds. A write that lands late still counts, from when it was sent, and a replica keeps writing every second while earlier writes are still on their way. Its membership writes have a NATS connection of their own, the lease connection (the same user, credentials and TLS as the broker's other NATS connection), so they never wait behind the replica's other NATS traffic. What that buys, and where it ends, is the supported NATS latency: up to about 6 s per NATS round trip no replica fences; from about 8 s replicas fence repeatedly. Only a NATS that stops answering, or answers slower than that, fences a replica: the one cut off, or all of them when NATS itself is down or that slow.
- A slow NATS does not hold up deliveries for authorization. Each subscriber's permission to read a topic is checked with the authorization service and remembered for
wsRevocationSLA(60 s by default). A remembered permission still in use is checked again in the background during the last third of that time, so the next value does not wait for the answer; a delivery that does have to wait for one (after a permission change, or a pause longer than that time) starts the checks of the subscriber's other recent topics at the same time. After a reconnect the subscriber starts with nothing remembered, so the first value of each topic waits for its own check, one topic after another (with NATS 2 s slow, 8 topics took about 30 s). A revoked permission still stops delivery withinwsRevocationSLA. Before, every value whose permission had just run out waited for its check, one topic after another: with each NATS reply 4 s late a persistent session on 8 topics was delivered about one value every 8 to 10 seconds. - Only a replica with a current lease decides for others. A replica that is fenced, or whose last write that landed was sent more than 10 seconds ago, never takes another replica for gone and never runs the elected sweep (expired sessions, overdue wills, the wills and client IDs of a gone replica): what it reads of NATS may be what its own writes do not reach. It gives the sweep up to a replica whose lease is current, and the 5-second absence of another replica counts again from when its own lease is current.
- Readiness. A replica that is fenced, cannot reach the shared stores for 15 seconds, runs under another process's replica ID, or reads less than the format floor answers
503on/health/ready, so Kubernetes takes it out of the Service and stops sending it connections. It is not restarted, and is ready again as soon as the cause is gone. When NATS is unreachable this is every replica at once: no replica serves, and the MQTT Broker page says No replica answered (see No broker replica answers).
What survives what
The table assumes two or more UNS replicas and a three-server NATS cluster (the production values). With one UNS replica, any loss of that replica is an MQTT outage until it is back.
| Event | Devices | Persistent sessions | Live subscribers |
|---|---|---|---|
| A UNS pod is stopped (rolling restart, drain) | Disconnected; reconnect to another replica. Wills wait for the reconnect grace (30 s). | Taken up on the replica the device reconnects to, with their subscriptions and what was kept for them. | Those on the stopped pod reconnect; values published meanwhile reach them only if their session is persistent. |
| A UNS pod is killed (crash, node loss) | As above. The others find it gone once its member entry has expired and stayed missing for 5 s (15 to 17 s after its last write); its wills wait the grace from then. | As above. A QoS 1 value it had delivered but whose acknowledgment was not yet stored may be delivered again (at least once). | As above. |
| One NATS server is lost | Not affected. A replica whose connections to it close reconnects to another server at once. One that hangs without closing them: a replica whose state connection is on it receives nothing from NATS on that connection, and once that has lasted 15 s it fences itself within the next 3 s (its devices reconnect to the others) and reconnects the connection to another server. If it was writing on that connection when the server stopped answering, the reconnect waits until that write gives up, at most a minute (the NATS client's write timeout); it is a member again a few seconds after the reconnect. Its membership is kept meanwhile (it has a connection of its own), so the others do not take it for gone. | Not affected: each stream keeps its other copies. Appends wait out a stream leader election, retried for up to 15 s. | Not affected. |
| Every NATS server is lost, or NATS is unreachable | Every replica fences itself within about 12 s: all MQTT connections are closed and refused until NATS is back. Their wills wait for the devices: the reconnect grace counts from when a replica serves again, so a device back as soon as the cluster is never has its will sent, and one that stays away has it sent once. | Kept in NATS, if its servers come back with their volumes; taken up again then. Lost with the volumes otherwise (see backup). | No live values until NATS is back. |
| NATS is slow but answering | Up to about 2 s per NATS round trip: full service. Up to about 6 s: no replica fences and nothing is lost, but devices wait: their publishes are acknowledged late or not at all, and from about 4 s most CONNECTs are refused "server unavailable" (their credential check cannot answer in time) and the devices retry. From about 8 s the replicas fence themselves repeatedly: the broker steps aside, and nothing it acknowledged is lost. See Supported NATS latency. | Not affected below about 6 s, apart from latency; above it, as for a whole-cluster outage while it lasts. | Not affected, apart from latency. |
| A UNS pod is cut off from NATS | Its devices are disconnected after about 12 s and reconnect to the other replicas. | Taken up by the other replicas. | As devices. |
| The mesh between two UNS pods is cut | Not affected. | Not affected: they read their session log in NATS. | Values published on one side miss subscribers on the other. Up to 64 MiB per other replica is buffered and sent when the link is back; past that the oldest are lost (MaestroHubUNSBrokerMeshValuesLost). |
| A rolling upgrade | As for a stopped pod, one replica at a time. | Kept. Replicas of the old and new release share formats both read; newer formats are written only once the last old replica is gone. | As for a stopped pod. |
Values devices publish are stored in the historian, which none of these events affects: whatever a device or subscriber missed live can be read from history.
Supported NATS latency
Every replica waits on NATS: its membership, every CONNECT (the credential check, the claim of the client ID and the take-up of a saved session) and every persistent session's values. How the cluster behaves depends on how long NATS takes to answer one request, its round trip:
| NATS round trip | What the cluster does |
|---|---|
| Up to about 2 s | Full service. CONNECTs fit their budget; values are acknowledged and delivered, a few seconds late. |
| Up to about 6 s | No replica fences and nothing is lost. Devices wait or retry: from about 4 s their publishes stall unacknowledged, and most CONNECTs are refused "server unavailable" (0x88; MQTT 3.1.1: 3), counted as credential_check_unavailable, because the credential check cannot answer in time. The devices retry, and are taken in once NATS answers faster. A device turned away like this is back, not gone: its will is not sent while the broker keeps refusing it. |
| Above about 6 s | The broker steps aside safely. The replicas fence themselves, repeatedly while it lasts: every MQTT connection is closed and new ones are refused. Nothing the broker acknowledged is lost, and recovery is correct: once NATS answers in time again, a device that comes back finds its saved session (within its session expiry interval) with every value it is owed, and its will is not sent. A device that stays away has its will sent once. |
Measured on the Kubernetes chart (three UNS replicas, a three-server NATS cluster, a soak of devices, persistent sessions and wills), with NATS slowed two ways: a slow network (tc netem on every NATS server) and a slow NATS (a proxy that holds every NATS reply):
- A steady 2, 4 and 6 s fenced no replica, in both models.
- From 4 s the devices' acknowledgements stalled and their CONNECTs were refused
credential_check_unavailable, about 200 for each two-minute step, and retried. - A steady 8 and 10 s fenced the replicas repeatedly: 17 and 14 fences in a two-minute step with netem, 12 and 12 with the proxy.
A sudden jump costs up to two round trips, once. The tiers above hold for a latency NATS reaches step by step, or holds steadily. When NATS goes from fast straight to a round trip of D, what each NATS server sends in the first D after the jump reaches the replicas about 2D after it, and every reply after that takes D. This is how TCP behaves on the NATS servers, not something a client can avoid: the server's retransmission timer still expects the fast round trip, so it takes the first reply for lost and sends nothing more on that connection until that reply is acknowledged, a round trip later. A jump from fast to more than about 5.3 s can therefore fence each replica once, about 10 to 12 s after the jump; the replicas are back a round trip later. Measured: a jump from no delay to 6 s on all three NATS servers fenced each replica once; 2, then 4, then 6 s fenced none. Nothing is lost either way.
Sites that expect such latency (a NATS cluster stretched over a slow WAN, for example) raise brokerMemberTTL: the membership timing scales with it (see Membership timing), so at 20 s the bound of each membership write doubles (see the table there); measure the latency your site sustains before relying on it. Its cost is failover: a replica that died is found gone, and its devices' wills and sessions are taken over, proportionally later: about 30 to 34 s after its last write at 20 s, against 15 to 17 s at the default. Raise connectBudget with it, so a CONNECT fits the slower round trips. A larger TTL does not make the devices' own wait shorter: each CONNECT and each acknowledged value still costs its NATS round trips.
Saved sessions during a whole-cluster outage. A persistent session expires its session expiry interval after its connection closed (the interval the client asked for in its CONNECT, capped by maxSessionExpiry), whether or not the client could reconnect meanwhile, counted from when the broker closed it, which a device may notice later (in the netem retest a device noticed 50 s after the broker had closed its connection, so it counted 9 min 47 s away where the broker counted 10 min 35 s). This is MQTT's rule, and the broker keeps it in every outage. When every replica is fenced, or refuses CONNECTs because NATS answers too slowly, for longer than a session's expiry interval, that session expires during the outage. When the device is back it gets a new session (CONNACK session present = 0) and must subscribe again. The values that waited for the old session in its session log are not delivered, even those the publishing devices had been acknowledged for: they belonged to the session that expired. They are still in the historian. Devices whose sessions must outlast an outage of NATS ask for a longer session expiry interval, for example an hour, and maxSessionExpiry must allow it. Its cost is the session's subscriptions and log position kept for that long after the device is gone for good. In the slownats retest, sessions with an interval of 600 s did not survive a 12¾-minute outage, and the one with 3,600 s did, with every value.
Clocks. The cluster assumes the clocks of the UNS pods and of the NATS servers do not jump. Two jumps break that, each once:
- A NATS server's clock stepping forward makes the member entries it holds expire early, by the size of the step. A replica connected to NATS notices at once and fences itself before the others act. A replica that is cut off from NATS at that moment cannot notice: if the step is larger than about 3.3 s (a third of the entry lifetime), the others may take it for gone, and send its devices' wills after the reconnect grace, while it still serves them, until it fences about 11.7 s after its last write.
- A UNS pod's own clock stepping forward by more than about 10.7 s makes it believe its membership has lapsed: it fences itself once (its devices reconnect elsewhere) and is a member again with its next write, a second later. A pod resumed after its VM was suspended for that long is fenced the same way, which is correct: the others may have taken it for gone.
Keep the nodes' clocks synchronised by NTP slewing rather than stepping (the default of chrony and systemd-timesyncd once they are in sync).
Rolling upgrades have one more rule: once the replicas have switched to a newer format, the cluster's format floor rises, and a release that cannot read it can no longer join. Roll back before the last old replica is replaced. See Rolling upgrades and When a replica is below the format floor.
A single node (the single binary, Docker Compose, the single-node chart) runs one broker with no cluster. Its persistent sessions are kept in memory, so a restart ends them: clients reconnect to new sessions and receive nothing published while they were away. See Persistent sessions.
What each QoS guarantees
Devices publishing to MaestroHub
| QoS | Guarantee |
|---|---|
| 0 | At most once. No acknowledgment. A value in flight when its replica stops is lost. A refusal is reported only on the device's _errors topic and in the reject feed. |
| 1 | At least once. PUBACK only once the value is stored in the historian or refused, never before. If the replica stops before that, the device holds the value and sends it again; a value whose PUBACK was lost may be stored twice. |
| 2 | PUBREC only once the value is stored or refused. For a persistent session, once every replica runs a release that keeps QoS 2 flows in the cluster, the packet ID is recorded after the value is stored and before the PUBREC: from the moment the device holds a PUBREC, the message is forwarded once, also if the session resumes on another replica. Before that point it may be forwarded twice, never lost: a replica that dies while the value is being stored leaves no record, and the device's resend is stored on the replica its session resumes on. |
In Enterprise, a QoS 1 or 2 value that a persistent session wants is also appended to its workspace's session log before the device's acknowledgment. When that log cannot take it within brokerSessionLogAppendWait, the publish is refused rather than acknowledged and lost.
Each replica decides which values a persistent session wants from its own copy of every session's subscriptions, which follows NATS a moment behind. While that copy is behind (its watch on NATS broke and is catching up, after a NATS server restart or a stream leader change), a device's value it does not know to be wanted waits for it, its acknowledgment with it, and is then kept or released; still behind after brokerSessionLogAppendWait, the publish is refused. A persistent session's SUBACK waits until every other replica's copy has its subscriptions (or that replica has been told of them over the mesh and decides nothing on its copy until it has them), for at most 2 s. So a device's value published on any replica after the SUBACK is kept for the session, or refused (never acknowledged and lost) when that replica's copy cannot catch up in time: this holds even when a replica's connection to NATS has gone half-open and its copy is frozen without knowing it. It needs the mesh: without one, or when a replica's mesh link is down for those 2 s, the SUBACK goes anyway (counted in maestrohub_uns_broker_subscribe_index_wait_timeouts_total), and a value published on that replica in the moment after it may not be kept for the session; a replica whose NATS connection receives nothing at all takes itself out of service after 15 s.
MaestroHub publishing to subscribers
Every stored value is published again on its topic for live subscribers at the QoS its device published it at, and at least liveMirror.qos (1 by default, 0 or 1): a value from the REST API, a pipeline or a connector at liveMirror.qos, a device's QoS 2 value at QoS 2. A subscriber receives the lower of that and the QoS it subscribed at. See Delivery guarantee of live values.
| Subscriber | What it gets |
|---|---|
| Clean session, QoS 0 | Each value at most once, while connected. |
| Clean session, QoS 1 | Each value at least once while connected. On disconnect, what it had not acknowledged is gone with the session. |
| Clean session, QoS 2 | A device's QoS 2 value exactly once while connected, on any replica. Other values at their lower QoS. On disconnect, what it had not completed is gone with the session. |
| Persistent session, QoS 2 (Enterprise) | A device's QoS 2 value exactly once, live or from the session log, whichever replica it resumes on and whichever replica dies (see below). While away, as for QoS 1. |
| Persistent session, QoS 1 (Enterprise) | While away: the QoS 1 and 2 values it wants are kept in its workspace's session log, for maxSessionExpiry and up to brokerSessionLogMaxBytes. When it reconnects, on any replica, it gets them in order from where it had read, then live values. Values the log dropped before it read them are missed and counted (MaestroHubUNSBrokerSessionLogValuesMissed). |
| Persistent session (single node) | Kept in memory for maxSessionExpiry, up to 8,192 + maxSubscriberQueue values per session; lost on restart. |
What exactly once covers
A device that publishes a value at QoS 2 into its inbox, and a subscriber of the value's topic that subscribed at QoS 2, get exactly once end to end:
- Covered: from the moment the device holds the PUBREC, the value reaches the subscriber once (live on the subscriber's replica or over the mesh from another, or from its workspace's session log after a persistent session resumes on any replica), also when the device's replica or the subscriber's replica dies at any step of either QoS 2 handshake (the device's PUBLISH, PUBREC, PUBREL, PUBCOMP; the delivery's PUBLISH, PUBREC, PUBREL, PUBCOMP). Each delivery to a persistent session is recorded in the cluster before its PUBLISH is written, so the replica the session resumes on completes it under the same packet ID instead of sending the value again.
- Not covered, at least once instead: a replica that dies after storing the value and publishing it again, but before recording the device's packet ID: the device has no PUBREC, sends the value again, and a subscriber may get it twice. Nothing is lost. A reading without its own
timestampis then also stored twice (see Timestamps and resends). - Not covered, can be missed: a clean-session subscriber whose replica dies (its session goes with the connection); a value its subscriber's queue drops (below); a value the session log dropped at its size cap before a persistent session read it (counted, see the table above).
A device value at QoS 0 or 1, and any value from the REST API, a pipeline or a connector, keeps the guarantee of its QoS above.
A slow subscriber drops its oldest values, whatever its QoS. When a subscriber cannot keep up and maxSubscriberQueue values wait for it, the oldest queued one is dropped for each new one. It stays connected and keeps receiving the newest values; the drops are counted in maestrohub_uns_broker_messages_dropped_total{reason="queue_full_oldest_dropped"}.
Limits
Every limit is per replica and read at start: change it and restart the UNS pods. On the Kubernetes chart a key left null keeps the broker's default. A value out of range stops the UNS service from starting, with an error naming the key and the allowed range.
In config.yaml of the single binary the keys are under modules.uns; the chart renders the same keys into each UNS pod's configuration.
embeddedBrokerLimits
Helm path: uns.mqtt.embedded.limits.<key>. config.yaml: modules.uns.embeddedBrokerLimits.<key>.
Connections
| Key | Default | Range | What it bounds |
|---|---|---|---|
maxClients | 5000 | at least 1 | MQTT clients connected at once. Next CONNECT: server busy (MQTT 3.1.1: server unavailable). Needs a file-descriptor limit above it. |
maxPendingConnections | 2048 | 1 to 1,000,000 | Connections between accept and CONNACK. At the limit the oldest one that has sent no CONNECT is closed. |
maxConnectsAuthenticating | 256 | 1 to 1,000,000 | CONNECTs in their credential check at once. Past it: server busy, no credential checked. |
maxConnectionsPerCredential | 1000 | 1 to 1,000,000 | Connections one credential may hold on a replica. Past it: quota exceeded (MQTT 3.1.1: server unavailable). |
trustClientAddresses | false | true or false | The source addresses the broker sees are the devices' own (no load balancer or NAT hides them). Only then do maxConnectionsPerSource and the failed-authentication throttle apply. |
maxConnectionsPerSource | 200 | 1 to 1,000,000 | Connections one source address may hold on a replica, with trustClientAddresses. |
maxConnectPacketBytes | 65536 | 1024 to the packet limit | The largest CONNECT, read before the client authenticated; also the largest Last Will. |
connectTimeout | 5s | 1s to 60s | How long a new connection may take to send its CONNECT. |
connectBudget | 20s | 5s to 120s | How long a CONNECT may take from its arrival to its CONNACK. The credential check (at most half of it), the claim of the client id across replicas and the take-up of the saved session share it, and each waits on NATS. Measured on real NATS, a clean session costs 3 NATS round trips and a persistent one resumed 4 (a new persistent session, or one with a will, 5), and the credential check one more, so the default fits a clean CONNECT while NATS answers within about 5 s and a persistent resume within about 4 s. Raise it when NATS answers slower, and keep it under what your devices wait for a CONNACK. A CONNECT past it is refused "server unavailable" (0x88; MQTT 3.1.1: 0x03) and the device retries. |
maxKeepAlive | 15m | 10s to 65535s | The longest an MQTT 5 connection may stay silent (given in its CONNACK). MQTT 3.1.1 clients keep their own. |
Failed authentication (applies only with trustClientAddresses)
| Key | Default | Range | What it bounds |
|---|---|---|---|
authFailureThreshold | 10 | 1 to 1,000,000 | Failed authentications of one user name from one address, within the window, that trip the throttle. |
authFailureWindow | 60s | 1s to 24h | The window the threshold counts in. |
authFailureBackoff | 30s | 1s to 24h | How long a tripped address and user name are refused; doubles with each trip. |
authFailureBackoffMax | 15m | 1s to 24h, at least authFailureBackoff | The longest that refusal grows to. |
Subscriptions and topics
| Key | Default | Range | What it bounds |
|---|---|---|---|
maxSubscriptionsPerClient | 1000 | 1 to 1,000,000 | Subscriptions one client may hold. Past it: SUBACK 0x97 (MQTT 3.1.1: 0x80). |
maxTopicLevels | 32 | 8 to 1024 | Levels in a network client's topic or filter. |
maxTopicBytes | 1024 | 256 to 65535 | Bytes in a network client's topic or filter. |
Delivery and sessions
| Key | Default | Range | What it bounds |
|---|---|---|---|
receiveMaximum | 1000 | 1 to 65535 | Unacknowledged QoS 1 and 2 publishes one client may have in flight to the broker. |
maxClientWritesPending | 1000 | 1 to 1,000,000 | Messages waiting for one subscriber's socket writer (8 bytes per slot, per connection). |
maxSubscriberQueue | 10000 | 1 to 1,000,000 | Messages waiting for one subscriber that cannot take more; past it the oldest is dropped. |
maxSessionExpiry | 1h | 0 to 168h, whole seconds | How long a persistent session is kept after its client disconnects. 0: none kept. In Enterprise also how long the session log keeps a value. |
maxOfflineSessions | 100 | 1 to 1,000,000 | Persistent sessions kept for disconnected clients at once. |
The Helm chart's values.yaml repeats each key with its default. More detail on the first six, with worked memory figures: Capacity limits.
The cluster's own settings
config.yaml key | Helm path | Default | What it does |
|---|---|---|---|
modules.uns.brokerSessionLogMaxBytes | uns.mqtt.embedded.sessionLogMaxBytes | 256 MiB (268435456); 0 means the default; at least 1 MiB | Caps each workspace's session log; the oldest values go first. JetStream reserves it for every workspace with persistent sessions, on every NATS server. |
modules.uns.brokerSessionLogAppendWait | uns.mqtt.embedded.sessionLogAppendWait | 10s (0 or null: the default) | How long a device's value waits for room in its session log, or for the log to open, or for the replica's copy of the persistent sessions to catch up, or for the retained store, before its publish is refused. |
modules.uns.brokerWillStopGrace | uns.mqtt.embedded.willStopGrace | 30s (0 or null: the default) | How long the wills of a stopping, fenced or gone replica's connections wait for their devices to reconnect, counted from when the replica stopped, fenced or was found gone; a will with a longer Will Delay Interval waits that. A device that reconnects to any replica within it does not have its will sent. A will of a connection on another replica is never sent by the new connection's replica before it is due: its own replica decides it. |
modules.uns.brokerMemberTTL | uns.mqtt.embedded.memberTTL | 10s (0 or null: the default); 5s to 2m | The membership entry lifetime; the rest of the timing scales with it (see Membership timing). Raise it only where NATS round trips of more than about 6 s are expected (see Supported NATS latency): a larger value keeps the replicas serving through a slower NATS, and makes failover slower by the same factor (a dead replica is found gone 1.5 to 1.7 TTLs after its last write). Every replica must use the same value: to change it, scale the UNS replicas to 0, remove the mqtt-members bucket (nats kv rm mqtt-members), then scale them back up; a replica started with a different value than the bucket's refuses to start. |
modules.uns.brokerMesh.listen | uns.mqtt.embedded.mesh.port | :7884 | The mesh listener. The chart also generates the mesh certificate (uns.mqtt.embedded.mesh.certDurationDays, 3650). |
modules.uns.liveMirror.qos | not a chart value | 1 | Lowest QoS of the live copy of a stored value, 0 or 1: the QoS of values from the REST API, pipelines, connectors and QoS 0 devices. A device's QoS 1 or 2 value keeps its QoS. |
These are fixed in this release and not configurable: store probe every 2 s with a 15 s bound before not ready, route write before a SUBACK 2 s, a persistent session's SUBACK waiting for the other replicas' copy of its subscriptions 2 s, mesh buffer 64 MiB per other replica, and up to 64 MiB of values per workspace waiting to be appended to its session log.
Membership timing
Every figure follows from the membership TTL (brokerMemberTTL, T). They are the timing rules; what the cluster sustains is measured, on Supported NATS latency:
| Rule | At the default (T = 10 s) | At T = 20 s | |
|---|---|---|---|
| A replica rewrites its entry | every T/10, whether or not earlier writes are back | 1 s | 2 s |
| Its entry expires | T after the last write NATS received | 10 s | 20 s |
| It fences itself | T × 7/6 after the sending of the last write that landed | 11.7 s | 23.3 s |
| The bound of one membership write | each write lands within T × 7/6 − T/10 (an upper bound: the replicas' other NATS traffic competes for the same servers; measured, no fence up to 6 s and repeated fences from 8 s at the default, see Supported NATS latency) | about 10.7 s | about 21.3 s |
| The others take a replica that died for gone | its entry missing for T/2 on their own clock: T × 1.5 to T × 1.7 after its last write | 15 to 17 s | 30 to 34 s |
| A replica decides for others only while | its last write that landed was sent less than T ago | 10 s | 20 s |
| A round-trip to NATS is given | T × 7/6 from its sending, cancelled when the lease lapses; once fenced, T × 7/6 | up to 11.7 s | up to 23.3 s |
| A fenced replica is a member again | once its connections are closed, it registers every T/10 whether or not earlier writes are back; the first that lands renews the lease from its sending, the next lands T/10 later | NATS within the bound above | the bound above |
| Its entry gone from NATS while its own lease is current (NATS expired it early, or it was deleted) | it writes it again and keeps its connections when that write lands within T/2 − T/3; otherwise it fences | 1.7 s | 3.3 s |
On the MQTT Broker page
The MQTT Broker page (Context Engine > Run, platform administrators) shows the cluster as the replica that answered its request sees it; its Cluster section is described in full on MQTT broker and historian.
- Replicas by pod name. Each replica is shown by its display name, the pod name (
maestrohub-uns-0). Hovering the name shows the full member ID, the pod name with the suffix of that start. - A restarted pod and a replica that stepped aside are different things. A pod that restarted is a new member under the same pod name: its tile says Restarted 12 s ago, and nothing is missing. A replica that left the cluster with no new member under its pod name has stepped aside: it takes no connections, its devices reconnect to the others, and the top line reads Degraded and links to A replica stepped aside.
- Links as seen from one replica. Only the answering replica knows its own mesh links, so the tiles of the others say whether they are linked to it, captioned Links as seen from <replica>. The next refresh may come from another replica and show its view. The per-pod gauges
maestrohub_uns_broker_cluster_membersandmaestrohub_uns_broker_cluster_mesh_links_up, and the alertMaestroHubUNSBrokerMeshLinksDown, cover every replica. - How many replicas there should be comes from the UNS pods (label
app.kubernetes.io/name=uns, ready or not), which the page reads only for someone who also holdsmonitoring.metrics:read. With it, the top line reads 2 of 3 replicas serving and a pod with no member shows as Not reporting with its status and restarts. Without it, the page counts only what the cluster knows: its members and the replicas that stepped aside. The replica count is not put into the UNS configuration: the StatefulSet carries a checksum of that configuration, so a scale change would restart every pod.
Settings the chart owns
On Kubernetes the chart owns every broker setting the page could change (deploymentOwns: [mode, listeners, tls, webSocket, publicHost], rendered into the UNS configuration). The page shows them as a read-only list, each with its Helm key, and a save or API call that changes one is refused with 409 BROKER_SETTING_MANAGED_BY_DEPLOYMENT (see Settings the deployment owns):
| Setting | Helm key |
|---|---|
| Mode | uns.mqtt.mode (empty: embedded, and a mode saved through the API before is kept) |
| MQTT port | uns.mqtt.embedded.tcpPort |
| Listen on | set by the chart: every interface (0.0.0.0) |
| WebSocket on/off and port | uns.mqtt.embedded.webSocket.enabled (null: off, and a WebSocket switched on at the page before is kept), uns.mqtt.embedded.webSocket.port |
| TLS certificate, port, mutual TLS, plaintext | uns.mqtt.embedded.tls.secretName, uns.mqtt.embedded.tlsPort, uns.mqtt.embedded.tls.clientCA, uns.mqtt.embedded.tls.disablePlaintext |
| Public address | uns.mqtt.service.publicHost (the load balancer's DNS name, a node's address for a NodePort); uns.mqtt.service.publicPorts (the ports devices use where the Service remaps them, any of mqtt, mqtts, ws, wss, for example {mqtts: 30883}) |
The list also shows the limits the broker runs with: Clients per replica (uns.mqtt.embedded.limits.maxClients), Longest session expiry (uns.mqtt.embedded.limits.maxSessionExpiry), Will grace after a replica stops (uns.mqtt.embedded.willStopGrace) and Membership TTL (uns.mqtt.embedded.memberTTL).
A change applies on the next rollout, one replica at a time. The public address is display only: no listener reads it, and nothing restarts for it. Without uns.mqtt.service.publicHost, Connect a device shows the Service's name inside the cluster and says it is reachable inside the cluster only. What a chart upgrade does with values saved on the page before is in the chart's UPGRADING.md (section MQTT broker settings: the chart owns them) and under Values kept from an earlier save.