Skip to main content
Version: 3.0
MaestroHub on Kubernetes · Tier 3

High availability on Kubernetes

Every module runs as its own service, most with two or more replicas, spread over three or more nodes. When a pod or a node fails, another replica takes over and MaestroHub keeps running.

Worker nodes
Three or more, 16 vCPU · 24 GB · 500 GB SSD each
Stateful services
NATS ×3, TimescaleDB ×3, Elasticsearch ×3
Licence
Enterprise
Run by your platform team

Your platform team installs, upgrades and runs this; MaestroHub supports the MaestroHub application. See Who is responsible for what.

What runs​

Each module is deployed on its own, with its own pods: no pod holds more than one module, unlike the single-node shape, where one pod holds them all. NATS, TimescaleDB and Elasticsearch run in pods of their own too. Kubernetes spreads each module's replicas over the nodes. The modules talk to each other through a three-server NATS cluster, so any replica can serve any request. Every UNS replica serves MQTT, and together they act as one broker at one address (how the clustered broker works).

High-availability installationBrowsers reach MaestroHub through your ingress controller, devices through the MQTT Service. Three nodes each run a replica of every module, a UNS replica serving MQTT, a NATS server, a TimescaleDB instance and an Elasticsearch node. The UNS replicas are linked by a TLS mesh; NATS, TimescaleDB and Elasticsearch replicate between the nodes. Your platform team provides the CloudNativePG operator, storage classes, an S3 bucket for backups, DNS and certificates, and Secrets.Browsers, APIUI and REST APIDevicesPLCs, gateways, sensorsKUBERNETES CLUSTERthree or more nodesIngress controlleryours · routes to UI and APIMQTT Service1883 · 8883 · one addressNODE 1MAESTROHUB MODULESuiauthauthzoauth2workspacesconnectorsengineschedulerwebsocketsearchlogssemanticsuns · MQTT brokera pod for each moduleSUPPORTING SERVICESNATS server 1TimescaleDB primaryElasticsearch node 1NODE 2MAESTROHUB MODULESuiauthauthzoauth2workspacesconnectorsengineschedulerwebsocketsearchlogssemanticsuns · MQTT brokera pod for each moduleSUPPORTING SERVICESNATS server 2TimescaleDB replicaElasticsearch node 2NODE 3MAESTROHUB MODULESuiauthauthzoauth2workspacesconnectorsengineschedulerwebsocketsearchlogssemanticsuns · MQTT brokera pod for each moduleSUPPORTING SERVICESNATS server 3TimescaleDB replicaElasticsearch node 3PROVIDED BY YOUR PLATFORM TEAMCloudNativePGruns TimescaleDBStorage classesvolumes for every nodeS3 bucketbackups and WAL archiveDNS, certificatesor cert-managerSecretspasswords and keys
Browsers, APIDevicesIngress controlleryoursMQTT Service1883 · 8883NODE 1 OF 3the other two run the sameMAESTROHUB MODULESuiauthauthzoauth2workspacesconnectorsengineschedulerwebsocketsearchlogssemanticsuns · MQTT brokermesh to the UNS replicas on nodes 2, 3SUPPORTING SERVICESNATS server↔ replicates to nodes 2 and 3TimescaleDB primary or replica↔ replicates to nodes 2 and 3Elasticsearch node↔ replicates to nodes 2 and 3PROVIDED BY YOUR PLATFORM TEAMCloudNativePGStorage classesS3 bucketDNS, certificatesSecrets
  • Requests: browsers and devices to any replica
  • Broker mesh: live MQTT values between UNS replicas
  • Replication: NATS, TimescaleDB, Elasticsearch
  • Run by your platform team

The drawing shows three nodes with a replica of each module on each. With the chart's production values most modules run two replicas, engine and WebSocket three, and App Studio (Beta) and the licence service one, so a failure of their pod stops them until Kubernetes starts it again. Raise the counts in the chart's values for your connections, tags and users.

What the cluster needs​

The chart deploys MaestroHub and the services it depends on. What it needs from the cluster is below; how your platform team provides it is up to them, so MaestroHub fits the tools you already run.

MaestroHub needs at least three schedulable worker nodes, each with 16 vCPU, 24 GB of RAM and 500 GB of SSD or NVMe storage for its volumes (12 vCPU and 16 GB at the least). That is the capacity MaestroHub uses, not the whole cluster: control-plane nodes and anything else the cluster runs come on top. Put each node on its own host, so one hypervisor or chassis cannot take out two of them.

Deployed by the chart

  • Every MaestroHub moduleUI, sign-in, connections, pipelines, scheduler, Unified Namespace with the MQTT broker, search, logs and the rest. With the production values most run 2 replicas and engine and WebSocket 3; App Studio (Beta) and the licence service run one each.
  • NATS with JetStream, 3 serversHow the modules talk to each other. Every stream and key-value bucket is kept on all three.
  • TimescaleDB, 3 instancesOne primary, two replicas and connection pooling, run through the CloudNativePG operator.
  • Elasticsearch, 3 nodesLog search and universal search.
  • Prometheus, alerts, log shippingScrapes every service; the alert rules ship with the chart.
  • Network policies and service accountsDefault-deny traffic rules and one service account per service.

Provided by your platform team

  • A storage class on SSD or NVMeDynamic provisioning of ReadWriteOnce volumes for NATS, TimescaleDB and Elasticsearch, with volume expansion allowed. The storage itself should be redundant.
  • Ingress controller and one addressFor example ingress-nginx, reachable on 443 from the user network. It must carry WebSocket and keep connections open for about an hour: live updates use them.
  • A DNS name and its certificateYour own certificate or cert-manager. Certificates between MaestroHub’s own services can be generated by the chart.
  • CloudNativePG operatorInstalled once in the cluster; MaestroHub provides the TimescaleDB image it runs.
  • A CNI that enforces network policiesCalico, Cilium or Canal, for example. Without it the chart’s default-deny rules do nothing.
  • A route from every node to your devicesConnections move to another node when one fails, so firewalls and device allow-lists must admit every worker node, not just one.
  • Synchronised clocksNTP on every node. Time-series timestamps come from the node clocks, and skew between nodes disturbs the NATS and database quorums.
  • Access to the container imagesPulled at installation, at upgrades and when a pod starts on a node that does not hold its image yet. A registry inside your network removes the need for internet access.
  • Kubernetes SecretsMade with kubectl, External Secrets with Vault, Sealed Secrets or SOPS: the chart reads ordinary Secrets. Keep a copy of the encryption keys outside the cluster.
  • An S3-compatible bucket, for backupsRecommended: the production values send nightly backups and the database’s write-ahead log there, so a restore does not depend on one node’s volume.
  • A load balancer or node ports for MQTTOnly when devices outside the cluster publish to the broker.

Not needed: a service mesh. Traffic between MaestroHub’s services already goes through NATS, with TLS in the production values.

When something fails​

Pick a failure to see where it lands on the three nodes and what it means for people using MaestroHub, for devices sending over MQTT and for stored data. The three dots on each failure are those three: petrol is not affected, amber is a reconnect or a wait, flame is no service until the cause is gone.

node 1modules+ uns brokernatsdb: primarysearchnode 2modules+ uns brokernatsdb: replicasearchnode 3modules+ uns brokernatsdb: replicasearch

What happens when you lose

A module replica

A pod crashes or is evicted

People and API clients

Keep working

The other replicas serve every request. Kubernetes starts a new pod.

Devices over MQTT

Reconnect

Devices on a UNS replica that went away reconnect to another one at the same address and take up their session there.

Stored data

Kept

Nothing that was acknowledged is lost. A connection owned by that replica is taken over by another one.

What MQTT clients keep through each failure, value by value, is on Broker cluster.

Security​

  • Restricted podsMaestroHub’s own services run under the Kubernetes restricted Pod Security Standard: not as root, no extra privileges. In the dependencies namespace, Elasticsearch, the log collector and node-exporter need access to the node, so that namespace must allow privileged pods.
  • Default-deny networkNetwork policies deny all traffic and allow only the paths MaestroHub uses.
  • TLS inside the clusterWith the chart’s production values, traffic between MaestroHub’s services and NATS, TimescaleDB and Elasticsearch is encrypted. The default values leave it off.
  • One service account eachEvery service has its own, with no access to the Kubernetes API.

Ports​

8080 · 8443

UI and API, HTTP and HTTPS

Your ingress controller, on 443

1883 · 8883

MQTT, MQTT over TLS

MQTT Service of the UNS replicas

8083

MQTT over WebSocket, when turned on

MQTT Service of the UNS replicas

The MQTT broker can be set to refuse plain MQTT, so devices connect over TLS only.

Upgrades​

An upgrade replaces the replicas one at a time while the others keep serving, so MaestroHub stays up. A release that converts stored data, or needs a short maintenance window, says so in its release notes.