Skip to main content
Version: 3.0 (next)

Back up and restore the broker's NATS state

What the clustered broker keeps in NATS JetStream, what each part is worth to devices, and how to save and bring it back.

Applies to
Kubernetes chart (clustered broker)
Tool
nats CLI
Not covered
UNS history (the historian has its own backups)

In an Enterprise deployment the UNS replicas keep the broker's shared state in NATS JetStream. Values devices publish are not part of it: they are stored in the historian, which is backed up with the databases. What JetStream holds is the broker's working state: sessions, retained messages, wills and the values kept for devices that are away.

A single node (the single binary, Docker Compose, the single-node chart) keeps this state in memory. There is nothing to back up there, and a restart ends its persistent sessions.

warning

Do not delete any of these streams or buckets while the UNS replicas run. A replica that loses a bucket or log reports itself not ready after 15 seconds, and one that cannot write its membership fences itself within about 12 seconds (at once when its member entry is deleted). A deleted bucket is created again when the UNS replicas restart.

What is stored, and what losing it means​

Each bucket is a JetStream stream named KV_<bucket>; each workspace's session log is a stream named LOG_mqtt-sessions-<workspace id>. nats stream ls lists them by those names.

StreamHoldsIf it is lostBack up?
KV_mqtt-session-stateEvery persistent session: its identity, subscriptions and expiry.Persistent sessions are gone. A device that reconnects finds no session (MQTT 5: session present 0) and must subscribe again; a device that does not resubscribe receives nothing.Yes
KV_mqtt-session-cursorHow far each persistent session has read its workspace's log, and the deliveries it had not acknowledged.Sessions keep their subscriptions but are fed live only: what was kept for them while away is not delivered.Yes
LOG_mqtt-sessions-<workspace id>The QoS 1 and 2 values a workspace's persistent sessions want, kept for maxSessionExpiry (1 hour by default) and up to brokerSessionLogMaxBytes.Sessions that were away miss what was published meanwhile. The values are in the historian: read them from history.Optional: the content ages out within maxSessionExpiry
KV_mqtt-retainedThe retained message of every topic that has one.Subscribers get no retained message on a topic until its device publishes one with retain again.Yes
KV_mqtt-session-willEach connection's Last Will, with the identity that declared it.Wills waiting to be sent (delayed, or handed over by a replica that stopped) are not sent.Yes
KV_mqtt-session-qos2Open QoS 2 flows of persistent sessions.Exactly once for QoS 2 is lost for the flows open at the restore: a QoS 2 message in flight when a session resumes elsewhere may be forwarded twice, a device's QoS 2 value to a QoS 2 subscriber included (see What exactly once covers).Optional
KV_mqtt-cluster-floorThe format floor: the oldest format a replica must read to join.The floor starts again from the lowest formats, so a replica of an older release could join and misread newer data.Yes
KV_mqtt-sessionsWhich replica and connection owns each client ID.Rebuilt as devices reconnect. The broker tolerates session states whose owner records are gone.No
KV_mqtt-routesWhich replica wants which topic filters.Rebuilt from each replica's subscriptions.No
KV_mqtt-membersThe live replicas. Entries expire 10 s after their last write.Rebuilt by the replicas' heartbeats. A restored copy would be expired anyway.No

The replicas also elect one of them to sweep expired sessions and gone replicas, through a leader lock (leader.mqtt-session-sweep) in their coordinator's bucket. It is re-elected on its own and needs no backup.

With global.natsAuth on (values-production.yaml), the broker's streams live in the BROKER account, which only the user uns-broker can use; no other service sees them. Without it, they are in NATS's default account with every other service's streams.

Connecting the nats CLI​

You need the nats CLI on a machine that can reach NATS, and the broker's credentials.

1. Credentials (with global.natsAuth on). The password of uns-broker is the key broker of the UNS service's NATS Secret, <global.natsAuth.secretName>-uns (default maestrohub-nats-auth-uns) in the app namespace:

kubectl -n maestrohub get secret maestrohub-nats-auth-uns \
  -o jsonpath='{.data.broker}' | base64 -d

2. Client certificate (with global.natsTLS on and mutual: true, as in values-production.yaml). The client certificate is in global.natsTLS.secretName (maestrohub-nats-client-tls) in the app namespace:

for k in ca.crt tls.crt tls.key; do
  kubectl -n maestrohub get secret maestrohub-nats-client-tls \
    -o jsonpath="{.data.${k//./\\.}}" | base64 -d > "$k"
done

3. Reach NATS. The chart's NATS Service is <release>-nats in the deps namespace, port 4222. From a workstation:

kubectl -n maestrohub-deps port-forward svc/maestrohub-nats 4222:4222

The generated server certificate names maestrohub-nats.maestrohub-deps.svc (and the shorter forms), not localhost. To verify it through the port-forward, map that name to 127.0.0.1 in your hosts file and connect to it by name.

4. A context:

nats context add mh-broker \
  --server nats://maestrohub-nats.maestrohub-deps.svc:4222 \
  --user uns-broker --password '<password from step 1>' \
  --tlsca ca.crt --tlscert tls.crt --tlskey tls.key \
  --select
nats stream ls

Leave out --user/--password without global.natsAuth, and the --tls* flags without global.natsTLS.

Back up​

A JetStream backup is a snapshot of one stream at one instant, taken over the network while the stream keeps running. The UNS replicas can stay up. The broker reads its logs with consumers it does not keep, so there are no consumers to save with the streams.

Take the cursors before the logs. A cursor that is older than its log only means a session gets a few values again after a restore (QoS 1 delivers at least once). A cursor that points past the end of its restored log makes the session read that log from its start.

DIR=broker-backup-$(date -u +%Y%m%dT%H%M%SZ)
mkdir -p "$DIR"

# 1. Cursors first, then the other buckets.
for s in KV_mqtt-session-cursor KV_mqtt-session-state KV_mqtt-session-will \
         KV_mqtt-session-qos2 KV_mqtt-retained KV_mqtt-cluster-floor; do
  nats stream backup "$s" "$DIR/$s"
done

# 2. The session logs, one per workspace with persistent sessions.
for s in $(nats stream ls --names | grep '^LOG_mqtt-sessions-'); do
  nats stream backup "$s" "$DIR/$s"
done

With global.natsAuth on, the BROKER account holds only the broker's streams, so nats account backup "$DIR" with the uns-broker context saves all of them at once (it does not order cursors before logs).

Keep the backup with your database backups, and take it at least as often as maxSessionExpiry: older session logs hold nothing a session can still use.

Restore​

Restore when NATS lost its JetStream data (its volumes were lost or reset) and you want devices' persistent sessions, retained messages and wills back. A restore recreates each stream with the messages, sequence numbers and configuration it had at backup time; anything written after the backup is not in it.

A stream can only be restored where it does not exist. The UNS replicas create every bucket when they start, so stop them first. MQTT is unavailable from step 1 to step 4.

  1. Stop the UNS replicas.

    kubectl -n maestrohub scale statefulset maestrohub-uns --replicas=0
    kubectl -n maestrohub wait --for=delete pod -l app.kubernetes.io/name=uns --timeout=5m

  2. Remove what they created since the loss, if anything. Only while the UNS replicas are stopped:

    nats stream ls --names | grep -E '^(KV_mqtt-|LOG_mqtt-sessions-)'
    nats stream rm <stream> -f      # for each stream you are about to restore

  3. Restore each stream you backed up:

    for d in "$DIR"/*/; do
      nats stream restore "$d"
    done

  4. Start the UNS replicas with your usual replica count:

    kubectl -n maestrohub scale statefulset maestrohub-uns --replicas=<n>
    kubectl -n maestrohub rollout status statefulset/maestrohub-uns

  5. Check. Every UNS pod is Ready; the log shows broker cluster: joined; the MQTT Broker page's Cluster row shows all replicas linked.

What devices see after a restore.

  • A persistent session reconnects to its restored state and is fed from its cursor. If its cursor was older than the log's end it gets some values again. If its cursor is not a position in the restored log, the replica logs broker cluster: a persistent session's cursor is not a position in its session log any more — ...; the session reads the log from its start and delivers the whole log to it.
  • A session whose cursor is older than the restored log's first value misses what lies between, counted in maestrohub_uns_broker_session_log_values_missed_total.
  • Values published between the backup and the loss are not in any log. Read them from history.
  • Start the UNS replicas on the release that took the backup, or a later one. A release that reads less than the restored floor refuses to start (see When a replica is below the format floor). Always restore KV_mqtt-cluster-floor together with the other buckets of the same backup, so the floor matches what they hold.

What is not verified here​

This page was written from the broker's code and from NATS's documentation. The commands were not run against a live cluster for this release. Check them on a test cluster before you depend on them:

  • The nats subcommands and flags differ between CLI releases. Recent releases also offer nats backup stream <stream> <dir> and nats backup restore stream <dir>; some older ones take the stream name on restore (nats stream restore <stream> <dir>). Run nats stream backup --help and nats stream restore --help for yours.
  • Whether a restored session log keeps its original creation time. If it does not, every cursor sees a new log and its session reads the log from its start (more values again, nothing lost).
  • nats account backup and nats account restore against the BROKER account.
  • The hosts-file mapping for TLS through a port-forward.