How the clustered MQTT broker works

In an Enterprise deployment on Kubernetes, every UNS replica runs MaestroHub's built-in MQTT broker, and the replicas act as one broker behind one address.

Status

Coming in the next 3.0 release

Applies to

Kubernetes chart, uns.mqtt.mode: embedded

Protocols

MQTT 3.1.1 and 5 (3.1 accepted) over TCP, TLS, WebSocket, WSS

Shared state

NATS JetStream, 3 copies of every stream

Coming in the next MaestroHub 3.0 release

The clustered broker described here is on its way and not in a released version yet. Today's broker, its settings and how it behaves on one node are on MQTT broker and historian.

This page is for whoever deploys and runs MaestroHub on Kubernetes: what the clustered broker gives you, how it is built, what happens on each failure, what each QoS level guarantees, how slow NATS may be, and the settings that change it.

What you get

• One address

  Devices connect to one Kubernetes Service, <release>-uns-mqtt, whichever replica ends up serving them.

• Any replica serves any device

  A value published on one replica reaches the subscribers of another in one hop, over the replicas' own TLS mesh.

• Nothing acknowledged is lost

  A QoS 1 or 2 publish is acknowledged only once the value is stored in the historian, or refused. A value is never acknowledged and then lost.

• Sessions survive failures and upgrades

  Persistent sessions live in NATS JetStream. After a pod crash or during a rolling upgrade, a device takes its session up on any replica, with what was kept for it.

• No false offline wills

  When a replica stops, its devices' wills wait 30 s for them to reconnect. A device back within that time has no will sent.

• No separate broker to run

  The broker is part of the UNS service: each UNS pod runs one broker node. There is no separate MQTT broker to deploy.

How it is built

There is no separate MQTT broker to deploy. The broker is part of the UNS service: each pod of the StatefulSet <release>-uns runs one broker node, and devices connect to the Service <release>-uns-mqtt. Any replica serves any client. The design splits the work in two:

• The data path: live values over the mesh. Live values travel replica to replica over the mesh, a direct TLS link between every pair of UNS pods. A value published on one replica reaches the subscribers of another in one hop. NATS does not carry them.
• The state path: what must be durable or agreed on, in NATS JetStream. Which replicas are members, which replica owns each client ID, every persistent session's state and read position, the per-workspace log of values kept for persistent sessions, retained messages, routes, QoS 2 flows, wills and the format floor.

Values devices publish are stored in the historian. The broker acknowledges a device's QoS 1 or 2 publish only once the value is stored (or refused), so whatever a subscriber missed live can always be read from history.

Scroll sideways to see the whole drawing.

• Data path: device to Service to replica
• Mesh: live values replica to replica
• Shared state connection
• Lease connection: membership writes only
• Stored values (drawn for one replica)

Each replica opens two NATS connections with the same user, credentials and TLS: one for the shared state, and the lease connection, which carries nothing but its own membership writes, so they never wait behind the replica's other NATS traffic.

What is kept where

Each bucket is a JetStream stream named KV_<bucket>; each workspace's session log is a stream named LOG_mqtt-sessions-<workspace id>.

Store	Holds	Back up
mqtt-session-state	Every persistent session: its identity, subscriptions and expiry.	Yes
mqtt-session-cursor	How far each persistent session has read its workspace's log, and the deliveries it had not acknowledged.	Yes
LOG_mqtt-sessions-<id>	The QoS 1 and 2 values a workspace's persistent sessions want, kept for maxSessionExpiry and up to sessionLogMaxBytes.	Optional
mqtt-retained	The retained message of every topic that has one.	Yes
mqtt-session-will	Each connection's Last Will, with the identity that declared it.	Yes
mqtt-session-qos2	Open QoS 2 flows of persistent sessions.	Optional
mqtt-cluster-floor	The format floor: the oldest format a replica must read to join.	Yes
mqtt-sessions	Which replica and connection owns each client ID.	No, rebuilt
mqtt-routes	Which replica wants which topic filters.	No, rebuilt
mqtt-members	The live replicas. Entries expire memberTTL after their last write.	No, rebuilt

The replicas also elect one of them, through a leader lock, to sweep expired sessions, overdue wills and replicas that are gone. With global.natsAuth on (as in values-production.yaml), all of this lives in the NATS account BROKER, which only the user uns-broker joins.

What happens when something fails

Pick an event to see what it means for devices, for persistent sessions and for live subscribers. The three dots on each event are those three audiences: petrol is not affected or kept, amber is a reconnect or a wait, flame is no service until the cause is gone. The explorer assumes two or more UNS replicas and a three-server NATS cluster. With one UNS replica, any loss of that replica is an MQTT outage until it is back.

What happens when

UNS pod stopped

Rolling restart, node drain

DevicesReconnect

Disconnected; they reconnect to another replica. Their wills wait for the reconnect grace (willStopGrace, 30 s): a device back within it has no will sent.

Persistent sessionsKept

Taken up on the replica the device reconnects to, with its subscriptions and what was kept for it.

Live subscribersReconnect

Those on the stopped pod reconnect. Values published meanwhile reach them only if their session is persistent.

What happens when

UNS pod killed

Crash, node loss

DevicesReconnect

As for a stopped pod. The other replicas find it gone 15 to 17 s after its last write; its wills wait the grace from then.

Persistent sessionsKept

As for a stopped pod. A QoS 1 value it had delivered whose acknowledgment was not yet stored may be delivered again.

Live subscribersReconnect

As for a stopped pod.

What happens when

One NATS server lost

Server or its node gone

DevicesNot affected

Connections to that server close and reconnect to another one. If it hangs without closing them, a replica whose shared-state connection is on it fences after 15 s of silence and is back a few seconds later; its devices reconnect to the others.

Persistent sessionsNot affected

Each stream keeps its other copies. Appends wait out a stream leader election, retried for up to 15 s.

Live subscribersNot affected

Not affected.

What happens when

Every NATS server lost

The whole NATS cluster down

DevicesRefused until back

Every replica fences within about 12 s: all MQTT connections are closed and refused until NATS is back. The will grace counts from when a replica serves again, so a device back as soon as the cluster is has no will sent.

Persistent sessionsKept

Kept in NATS if its servers come back with their volumes. A session whose expiry interval runs out during the outage expires, as MQTT requires.

Live subscribersPaused

No live values until NATS is back.

What happens when

NATS slow

Slow but still answering

DevicesWait or retry

Full service up to about 2 s per NATS round trip; devices wait or retry up to about 6 s; above that the broker steps aside safely. See the latency bands below.

Persistent sessionsNot affected

Not affected below about 6 s, apart from latency.

Live subscribersNot affected

Not affected, apart from latency.

What happens when

UNS pod cut off from NATS

Network cut

DevicesReconnect

Disconnected after about 12 s, when the cut-off replica fences itself; they reconnect to the other replicas.

Persistent sessionsTaken up

Taken up by the other replicas.

Live subscribersReconnect

As for devices: disconnected at the fence, they reconnect to the others.

What happens when

Mesh cut between two pods

Network cut between replicas

DevicesNot affected

Not affected.

Persistent sessionsNot affected

Not affected: persistent sessions read their session log in NATS.

Live subscribersPartly delayed

Values published on one side miss subscribers on the other. Up to 64 MiB per other replica is buffered and sent when the link is back; past that the oldest are lost and counted.

What happens when

Rolling upgrade

New release, one replica at a time

DevicesReconnect

As for a stopped pod, one replica at a time.

Persistent sessionsKept

Kept. Old and new replicas write only formats both read; newer formats are written once the last old replica is gone.

Live subscribersReconnect

As for a stopped pod.

Rollback limit: once the replicas have switched to a newer format they raise the cluster's format floor, and a release that cannot read it refuses to start. Roll back before the last old replica is replaced. Upgrade one release at a time: two releases with no mesh protocol version in common do not link.

Failover, second by second

Every figure follows from the membership TTL T (memberTTL, 10 s by default). The drawing shows the default: a replica that is still running but cut off from NATS, a replica that was killed, and the wills of the killed replica's devices.

Scroll sideways to see the whole timeline.

1. Heartbeat. Each replica rewrites its entry in mqtt-members every T/10 (1 s), over its lease connection, whether or not earlier writes are back. A write that lands late counts from when it was sent.
2. Fencing. A replica none of whose writes landed for T × 7/6 (about 11.7 s), counted from when that write was sent, fences itself: it closes every MQTT connection (MQTT 5 clients get DISCONNECT 0x8B, server shutting down), hands its devices' wills over, and refuses new CONNECTs (server unavailable) until a write of its entry lands again. One that finds its entry gone from NATS fences at once. Either way it stops serving before the others can act for its devices.
3. Dwell. The entry expires T after the last write NATS received. The other replicas take a replica for gone only once its entry has been missing for T/2 (5 s) more on their own clock: 15 to 17 s after its last write.
4. Wills. The wills of a stopped, fenced or gone replica's connections wait for the reconnect grace (willStopGrace, 30 s). A device back within it has no will sent.

Only a replica with a current lease decides for others. A fenced replica, or one whose last landed write was sent more than T ago, never takes another for gone and never runs the elected sweep.

Half-open connections. The store probes run every 2 s on the shared-state connection. If that connection receives nothing at all for 15 s (for example its NATS server stopped with the TCP connection left open), the replica fences itself, reconnects it, and registers again once it receives anything. Its lease stays renewed meanwhile, so the others do not take it for gone for this alone. Measured: fenced 17 s after the connection went silent, back 3 s after it answered.

	Rule	T = 10 s (default)	T = 20 s
Heartbeat	every T/10	1 s	2 s
Entry expires	T after the last write NATS received	10 s	20 s
Replica fences itself	T × 7/6 after the sending of the last write that landed	11.7 s	23.3 s
Bound of one membership write	T × 7/6 minus T/10 (upper bound; the measured envelope is smaller, see below)	about 10.7 s	about 21.3 s
A dead replica is taken for gone	entry missing for T/2: T × 1.5 to T × 1.7 after its last write	15 to 17 s	30 to 34 s
Decides for others only while	its last landed write was sent less than T ago	10 s	20 s

What each QoS guarantees

Devices publishing to MaestroHub

QoS 0At most once

No acknowledgment; a value in flight when its replica stops is lost. A refusal is reported only on the device's _errors topic and in the reject feed.

QoS 1At least once

PUBACK only once the value is stored in the historian or refused, never before. If the replica stops first, the device still holds the value and sends it again. A value whose PUBACK was lost may be stored twice: send a timestamp to have it stored once.

QoS 2Forwarded once

PUBREC only once the value is stored or refused. For a persistent session the packet ID is recorded in the cluster after the value is stored and before the PUBREC: from the moment the device holds a PUBREC, the message is forwarded once, wherever the session resumes.

A QoS 1 or 2 value that a persistent session wants is also appended to its workspace's session log before the device's acknowledgment, and a value on a retaining topic is acknowledged once the shared retained store has it. When either cannot take the value within sessionLogAppendWait (10 s), the publish is refused: MQTT 5 devices get PUBACK 0x80, MQTT 3.1.1 connections are closed after the acknowledgments before it, and the device sends the value again. A value is never acknowledged and then lost.

MaestroHub publishing to subscribers

Every stored value is published again on its topic at the QoS its device used, and at least liveMirror.qos (1 by default, see Delivery guarantee of live values). A subscriber gets the lower of that and its subscription QoS.

QoS 0

QoS 1

QoS 2

Clean session

Clean session, QoS 0Each value at most once, while connected.

Clean session, QoS 1Each value at least once while connected. On disconnect, what it had not acknowledged goes with the session.

Clean session, QoS 2A device's QoS 2 value exactly once while connected, on any replica.

Persistent session

Persistent session, QoS 0QoS 0 values are not kept for a session while it is away.

Persistent session, QoS 1While away, the values it wants are kept in its workspace's session log. On reconnect, on any replica, it gets them in order from where it had read, then live values. Values the log dropped at its size cap are missed and counted.

Persistent session, QoS 2A device's QoS 2 value exactly once, live or from the session log, whichever replica it resumes on and whichever replica dies.

What exactly once covers

Covered

A device publishing at QoS 2 to a subscriber at QoS 2: from the moment the device holds the PUBREC, the value reaches the subscriber once, live, over the mesh, or from the session log after a resume, also when either replica dies at any step of either QoS 2 handshake.

At least once instead

A replica that dies after storing and publishing the value but before recording the device's packet ID: the device has no PUBREC and sends again; a subscriber may get it twice. Nothing is lost.

Can be missed

A clean-session subscriber whose replica dies; a value its subscriber's queue drops (maxSubscriberQueue, oldest first); a value the session log dropped at its size cap before the session read it.

How slow NATS may be

Every replica waits on NATS: for its membership, for every CONNECT (the credential check, the claim of the client ID, the take-up of a saved session) and for every persistent session's values. How the cluster behaves depends on one NATS round trip:

0 s2 s6 sslower

1. Up to about 2 s per NATS round tripFull service

   CONNECTs fit their budget; values are acknowledged and delivered, a few seconds late.

2. Up to about 6 s per NATS round tripNo fence, devices wait or retry

   No replica fences and nothing is lost. From about 4 s publishes stall unacknowledged and most CONNECTs are refused server unavailable (0x88; MQTT 3.1.1: 3), counted as credential_check_unavailable. A device turned away like this has no will sent while it is refused.

3. Above about 6 s per NATS round tripSteps aside safely, then recovers

   Replicas fence themselves, repeatedly while it lasts. Nothing acknowledged is lost. Once NATS answers in time again, a device back within its session expiry finds its session with every value it is owed and has no will sent; a device that stays away has its will sent once.

Measured on the Kubernetes chart (three UNS replicas, a three-server NATS cluster, a soak of devices, persistent sessions and wills) with NATS slowed two ways, tc netem on every NATS server and a proxy that holds every NATS reply: a steady 2, 4 and 6 s fenced no replica in either model; a steady 8 and 10 s fenced the replicas repeatedly.

Sites that expect slower NATS (a NATS cluster stretched over a slow WAN) raise memberTTL. The membership timing scales with it, so at 20 s each membership write may take about twice as long; the cost is failover, 30 to 34 s instead of 15 to 17 s. Raise connectBudget with it so CONNECTs fit the slower round trips, and measure what your site sustains before relying on it.

Clocks

The cluster assumes neither the UNS pods' clocks nor the NATS servers' clocks step. Keep them synchronised by NTP slewing (the default of chrony and systemd-timesyncd once in sync).

Sign-in, tenancy and limits

• Sign-in. Each device signs in as its own OAuth2 API client: the client ID as MQTT user name and the client secret as password, or an OAuth2 access token from that client as password. It gets exactly that client's topic permissions. A session's credential is re-checked before its deadline; a revoked, disabled or rotated credential closes the session.
• Writes go through the inbox. A device publishes only to <version>/<workspace-slug>/_ingress/<its client id>/<topic>. Another device's inbox, another workspace, or a data-node topic is refused (MQTT 5: 0x87). The value then passes the same validation chain as a REST write: schema, permissions, quality, alerts, history. The same rule applies to Last Wills. The retain flag is cleared on inbox publishes. The inbox grammar, replies and rejects are on MQTT Write Endpoint.
• Reads are checked on every delivery path: live on the subscriber's replica, from the mesh, and from a session log. A subscriber's permission is remembered for at most wsRevocationSLA (60 s by default), so a revoked permission stops delivery within that time.
• Workspaces. Writes are scoped to the device's own workspace inbox and reads to its grants. Each workspace has its own session log, and a workspace whose log stalls holds only its own devices. Client IDs are broker-wide: a client ID held by another identity, in any workspace, is refused (MQTT 5: 0x85).
• NATS users. With global.natsAuth.enabled (on in values-production.yaml) every service signs in to NATS as its own user, and the broker's state is in the BROKER account, which no other service can read or write. Switching it on a running NATS cluster deletes the JetStream data; follow the chart's UPGRADING.md.
• TLS. MQTT over TLS on 8883 from a kubernetes.io/tls Secret, optionally with mutual TLS (clientCA) and with the plaintext listener closed (disablePlaintext). The mesh always uses TLS 1.3 with a certificate the chart generates per deployment; a link trusts no other.
• Connection and abuse limits are enforced per replica before and after authentication: clients, pending connections, CONNECTs authenticating at once, connections per credential and per source address, CONNECT and Will size, failed-authentication throttle, subscriptions per client, topic depth and length, in-flight window and subscriber queue. See Limits.

Configuration

Every setting is per replica and read at start: change it and restart the UNS pods. On the chart, a key left null keeps the broker's default; a value out of range stops the UNS service from starting, with an error naming the key and the range.

Cluster settings

Keys under uns.mqtt.embedded in the chart; the matching config.yaml key is named in each row.

Key	Default	What it does
memberTTL	10s (5 s to 2 m)	Membership TTL; the timing above scales with it. Same on every replica: to change it, scale UNS to 0, run nats kv rm mqtt-members, scale back up. A replica started with a different value than the bucket's refuses to start. config.yaml: modules.uns.brokerMemberTTL.
limits.connectBudget	20s (5 s to 120 s)	Time from a CONNECT's arrival to its CONNACK, shared by the credential check (at most half), the client ID claim and the session take-up. A clean CONNECT costs 3 NATS round trips, a persistent resume 4 (a new persistent session, or one with a will, 5), plus 1 for the credential check. Past it: server unavailable and the device retries. config.yaml: …embeddedBrokerLimits.connectBudget.
willStopGrace	30s	How long the wills of a stopped, fenced or gone replica's connections wait for their devices to reconnect. config.yaml: modules.uns.brokerWillStopGrace.
sessionLogAppendWait	10s	How long a value waits for room in its session log, or for the retained store, before its publish is refused. config.yaml: modules.uns.brokerSessionLogAppendWait.
sessionLogMaxBytes	256 MiB	Cap of each workspace's session log, oldest out first. JetStream reserves it for every workspace with persistent sessions, on every NATS server: raise nats.config.jetstream.fileStore.maxSize with it. config.yaml: modules.uns.brokerSessionLogMaxBytes.
mesh.port	7884	Mesh listener. Certificate in Secret <release>-uns-mesh-tls, valid mesh.certDurationDays (3650). config.yaml: modules.uns.brokerMesh.listen.

Deployment values around the broker:

Setting	Default	What it does
uns.replicas	1 (values-production.yaml: 2)	UNS replicas, each a broker node. More than one needs storage.type: timescale.
global.natsReplicas	3	Copies of every stream.
global.natsAuth.enabled	false (production: true)	Per-service NATS users and the BROKER account.
modules.uns.liveMirror.qos (config.yaml only, not a chart value)	1	Lowest QoS of the live copy of a stored value (0 or 1).

Fixed in this release: store probe every 2 s with a 15 s bound, route write before a SUBACK 2 s, mesh buffer 64 MiB per other replica, and up to 64 MiB of values per workspace waiting to be appended to its session log.

Listeners and Service

Helm path	Default	What it does
uns.mqtt.embedded.tcpPort / tlsPort	1883 / 8883	MQTT and MQTT over TLS.
uns.mqtt.embedded.webSocket.port	8083	WebSocket (WSS with a certificate); switched on in UNS settings.
uns.mqtt.embedded.tls.secretName	""	TLS Secret; renewals reach the next handshake without a restart.
uns.mqtt.embedded.tls.clientCA / disablePlaintext	false	Mutual TLS; TLS only.
uns.mqtt.service.type	ClusterIP	LoadBalancer or NodePort to reach devices outside the cluster.
networkPolicy.mqttFrom		Narrows who may connect, with networkPolicy.enabled.

Limits

Each key is under uns.mqtt.embedded.limits.<key>.

Key	Default	Bounds
maxClients	5000	Clients connected at once.
maxPendingConnections	2048	Connections between accept and CONNACK.
maxConnectsAuthenticating	256	CONNECTs in their credential check at once.
maxConnectionsPerCredential	1000	Connections one credential may hold on a replica.
trustClientAddresses / maxConnectionsPerSource	false / 200	Per-address limit and failed-auth throttle, only when addresses are the devices' own.
authFailureThreshold, Window, Backoff, BackoffMax	10, 60s, 30s, 15m	Failed-authentication throttle per address and user name.
maxConnectPacketBytes	65536	Largest CONNECT, read before authentication; also the largest Will.
connectTimeout	5s	Time a new connection may take to send its CONNECT.
maxKeepAlive	15m	Longest MQTT 5 keepalive.
maxSubscriptionsPerClient, maxTopicLevels, maxTopicBytes	1000, 32, 1024	Subscriptions and topic shape.
receiveMaximum, maxClientWritesPending, maxSubscriberQueue	1000, 1000, 10000	In-flight window and queues; a full subscriber queue drops its oldest.
maxSessionExpiry, maxOfflineSessions	1h, 100	Persistent sessions kept and for how long (also how long the session log keeps a value).

Readiness, metrics and alerts

• Readiness. A replica that is fenced, cannot reach the shared stores for 15 s, runs under another process's replica ID, or reads less than the format floor answers 503 on GET /health/ready (port 8080) with a reason starting broker-cluster-store:. Kubernetes stops sending it connections; it is not restarted, and is ready again once the cause is gone.
• Status. GET /api/v1/system/uns/settings/status (needs uns_settings:update) carries broker.cluster: replica, mesh, members, memberCount, linksUp, and when set storeUnreachable, fenced, duplicateReplicaId, belowFormatFloor, sessionLogValuesMissed, sessionLogReadsHeld. The MQTT Broker page reads it. Each answer is the replica's that served it.
• Logs. Every broker line starts with broker cluster:, mesh: or embedded broker:.

Alerts. With alerts.enabled the chart ships twelve broker alerts (three in maestrohub.uns-ingest, nine in maestrohub.uns-broker-cluster), each with a runbook_url to its runbook entry:

Alert	Means
MaestroHubUNSIngestStoreUnavailable warning	The historian did not answer ingest writes for a whole retry budget, or a write hung.
MaestroHubUNSIngestQuarantined warning	Incoming values were given up on after repeated hung writes.
MaestroHubUNSLiveMirrorDropping warning	Live MQTT copies of stored values are not published.
MaestroHubUNSBrokerMeshLinksDown warning	A replica has fewer mesh links than other replicas, for 5 minutes.
MaestroHubUNSBrokerMeshValuesLost warning	Live values from another replica never arrived (buffer past 64 MiB).
MaestroHubUNSBrokerSessionLogValuesMissed warning	Persistent sessions missed values the log dropped at its cap.
MaestroHubUNSBrokerSessionLogAppendsHeld warning	Acknowledgments wait for room in a session log.
MaestroHubUNSBrokerSessionLogAppendsRefused warning	Publishes were refused after sessionLogAppendWait.
MaestroHubUNSBrokerSessionLogAppendsReordered warning	Values stored in a session log ahead of an earlier value of their topic.
MaestroHubUNSBrokerSessionSubscriptionsKept critical	A state write tried to change stored subscriptions without a SUBSCRIBE; kept; report it.
MaestroHubUNSBrokerRoutesLate warning	Subscriptions acknowledged before the other replicas know them.
MaestroHubUNSBrokerClusterFormatUnreadable critical	A replica met cluster state in a newer format; finish the rolling upgrade.

Useful counters (prefix maestrohub_): uns_broker_cluster_members, uns_broker_cluster_mesh_links_up, uns_broker_cluster_members_gone_total, uns_broker_connect_refused_total{reason} (for example fenced, client_id_in_use, max_clients), uns_broker_messages_dropped_total, uns_broker_wills_withheld_total, uns_broker_subscribe_index_wait_timeouts_total.

MQTT features not offered yet

Feature	Today	Status
Shared subscriptions ($share/…)	CONNACK announces them as not available; a $share SUBSCRIBE is refused.	Futureplanned, delivered over the mesh
Subscription identifiers	Announced as not available; a SUBSCRIBE carrying one is refused (0xA1).	Buildingnext release
Topic aliases	The broker accepts no client topic aliases (Topic Alias Maximum 0).	Buildingnext release

A single node

The single binary, Docker Compose and the single-node chart run one broker with no cluster. Its persistent sessions are kept in memory, so a restart ends them: clients reconnect to new sessions. Every other guarantee on this page that does not involve a second replica holds there too: a device's value is acknowledged only once stored, and a QoS 2 device value reaches a QoS 2 subscriber exactly once. The single-node broker's settings are on MQTT broker and historian.

Related pages

• MQTT broker and historian: the broker's settings page, section The broker in Enterprise.
• Persistent sessions: how a session is kept and for how long.
• MQTT Write Endpoint: the inbox grammar, replies and rejects.
• OAuth2 clients: the credentials devices sign in with.