Sizing and performance
What one machine running Context Engine with its embedded MQTT broker and embedded time-series database stored and executed, how it was measured, and how to size a deployment from it.
- Measured
- 16 October 2026 · MaestroHub 3.0
- Deployment
- Tier 1, one machine
- Machines
- GCP C3 · 4, 8 and 22 vCPU
- Guaranteed ingest
- 29,852 · 59,905 · 164,376 values/s
Every figure on this page was measured on 16 October 2026 on MaestroHub 3.0, on Google Cloud C3 machines (c3-highcpu-4, c3-highcpu-8 and c3-highcpu-22, Intel Xeon Platinum 8481C). A figure holds for that version, that machine type and the workload described here. We run the same test again before every release, and we can run it with you on your own hardware.
The deployment measured is Tier 1: Context Engine with its embedded MQTT broker and embedded time-series database, on one machine. Enterprise (Kubernetes, several nodes) was not measured; see What the test does not cover.
The results
Measured 16 October 2026, MaestroHub 3.0, one Tier 1 machine per column. A vCPU is a logical core: each physical core runs two hardware threads.
c3-highcpu-4 | c3-highcpu-8 | c3-highcpu-22 | |
|---|---|---|---|
| Cores | 4 vCPU = 2 physical × 2 threads | 8 vCPU = 4 physical × 2 threads | 22 vCPU = 11 physical × 2 threads |
| Ingest, guaranteed (10-minute hold) | 29,852 values/s | 59,905 values/s | 164,376 values/s |
| Ingest, maximum (10-minute hold) | 39,865 values/s | 89,248 values/s | 198,982 values/s |
| What limited the maximum | CPU | CPU | CPU |
| Pass-through pipeline | 4,400 runs/s | 8,400 to 10,000 runs/s | 18,000 runs/s |
| Pipeline with 20 ms of script work | 86 runs/s | 170 runs/s | 480 runs/s |
| Pipeline that waits 20 ms | 4,400 runs/s | 8,000 runs/s | 10,000 runs/s |
| Pass-through pipeline, runs in order | 2,900 runs/s | 4,500 runs/s | 6,000 runs/s |
- Guaranteed is the rate to size by. Held for 10 minutes, every value was stored, publish to stored p99 stayed under 100 ms and MaestroHub's CPU stayed under 80% of its vCPUs.
- Maximum is the most the machine stored in full past that rule, held for 10 minutes. It is the point where the machine is full, not a rate to plan for.
- Stored before acknowledged. A device's value is acknowledged only once it is in the historian, so the publish to stored latency is also the time a device waits for its acknowledgement. See What an acknowledgement means.
- Pipeline rates are the most runs per second executed with no run lost, one class at a time with nothing else running. They are maximums, measured in steps of 40 to 60 seconds. Where two runs gave different results, both are shown.
The sections below describe the machines, the workloads and the method, then every step of the curves and what limited each result.
The test setup
c3-highcpu-4 | c3-highcpu-8 | c3-highcpu-22 | |
|---|---|---|---|
| CPU | Intel Xeon Platinum 8481C, 2.70 GHz | same | same |
| Logical / physical cores | 4 / 2 | 8 / 4 | 22 / 11 |
| Machine memory | 8 GB | 16 GB | 44 GB |
| MaestroHub memory cap | 4 GB | 8 GB | 16 GB |
| Data disk (Hyperdisk Balanced), provisioned | 100 GB, 10,000 IOPS, 300 MB/s | 100 GB, 15,000 IOPS, 500 MB/s | 200 GB, 20,000 IOPS, 800 MB/s |
| Data disk, measured with fio | 10,570 IOPS, 300 MB/s | 15,859 IOPS, 500 MB/s | 21,138 IOPS, 800 MB/s |
- MaestroHub had every vCPU of its machine. It ran as one process in a container pinned to all logical cores, with nothing else on the machine but the operating system, Docker and a small agent that read CPU, disk and network counters for the test. The memory cap is the container's limit, with the Go memory limit at 85% of it.
- All of MaestroHub's data was on the data disk: the historian, the execution history and the internal message streams. The boot disk carried only the operating system.
- The load generator ran on its own machine in the same zone (
europe-west4-b), so its CPU was never counted against MaestroHub: ac3-highcpu-8for the 4 vCPU machine and ac3-highcpu-22for each of the others. In the 10-minute holds at the maximum it was 29%, 21% and 47% busy on the 4, 8 and 22 vCPU runs. - Network between the two machines, measured with iperf3 before the runs: 21.1 to 21.6 Gbit/s, 0.1 ms round trip.
- Operating system: Ubuntu 24.04, Docker Engine with cgroup v2.
Configuration
MaestroHub ran with its default configuration and security on, except for the settings listed here.
- Security on. Every device signed in to the embedded MQTT broker as its own API client (OAuth2 client credentials), and every value was checked against that client's topic permissions. MQTT ran over plain TCP on port 1883, without TLS.
- Device ingest rate limits off.
modules.uns.ingress.perClientRatePerSecandmodules.uns.ingress.perWorkspaceRatePerSecwere0. Their defaults (2,000 values/s per device and 20,000 values/s per workspace) would have capped the test below the machine. See Settings that change the numbers. - History kept 32 days for every topic.
- Logging at
warn, the web UI off, and the Prometheus/metricsendpoint on, so the load generator could read MaestroHub's counters. - Live re-publish at QoS 1 (
modules.uns.liveMirror.qos: 1, the default): every stored value was published again on its topic. - Pipeline execution history at Standard, the default level.
- The historian's disk writes at their default (
historian.pebble.syncWritesoff): see What an acknowledgement means. - The test build lifted the evaluation licence's cap of 10 pipelines, so the pipeline tests could deploy hundreds.
The build was MaestroHub 3.0, from the 3.0 development branch on 16 October 2026 (commit e994324ae5).
The workloads
Device ingest
- 1,000 devices × 10 tags = 10,000 topics. Each device is an MQTT 5 client with its own connection.
- QoS 1, one value per message. A message is a JSON envelope with one number, for example
{"value": 42.7}, published into the device's inbox (mHv1.0/<workspace>/_ingress/<client id>/<path>), the way a plant device writes through the MQTT write endpoint. - Each device keeps at most 16 messages in flight, sent and not yet acknowledged.
- The first value on a new path creates its topic. Topic creation happens in a warm-up before the measured steps.
- No pipelines run in the ingest test.
A value counts as stored when MaestroHub has done everything it does for a value written in production:
- authenticated the device and checked its permission to publish on the topic,
- checked the value against the topic,
- written it to the embedded time-series database (the historian),
- published it live on its topic in the Unified Namespace.
One message in 100 asks for its verdict on an MQTT 5 response topic; the time from publishing to that verdict is the publish to stored latency. After each run, the load generator read 40 topics back from the historian, value by value, and compared them with what the devices had sent and had acknowledged.
Pipelines
Every pipeline is created through the same API call the designer uses, and is started the way a plant starts one: a device publishes a value, MaestroHub stores it, and a UNS trigger on that topic starts the pipeline. The pipeline's last node publishes a derived value on a twin topic, so each run stores two values, the device's and the derived one. The latency reported is from the device's publish to the derived value arriving at a live subscriber.
| Class | Nodes |
|---|---|
| Pass-through | trigger.uns.subscribe → transform.set (adds one field) → system.uns.publish |
| 20 ms of script work | trigger.uns.subscribe → logic.javascript (an arithmetic loop calibrated on the machine to about 20 ms) → system.uns.publish |
| Waits 20 ms | trigger.uns.subscribe → action.wait (20 ms) → system.uns.publish |
| Pass-through, in order | as pass-through, with Run order set to one at a time, in arrival order |
- Class runs: 20 pipelines, fed by 100 devices × 4 tags, each pipeline on the tags of 5 devices.
- Beside ingest: the pass-through and script classes again, while 200 other devices stored 65% of the machine's guaranteed rate (19,500, 39,000 and 107,250 values/s) on topics no pipeline subscribes to.
- A fleet: 1,000 and then 2,000 pipelines, each on a tag of its own and each run once a second.
- Every pipeline ran at Normal priority, with Step execution Sequential and Execution History Detail Standard.
How the limits were found
Ingest. The rate steps up, each step held for one minute: 5,000, 10,000, 20,000, 30,000 and 40,000 values/s, then 20% more per step until the machine no longer stores what is offered (less than 99% of the offered rate stored in the step). A step that fails is run again once before it counts. Then:
- the highest step that meets the guaranteed rule (every value stored, publish to stored p99 under 100 ms, MaestroHub CPU under 80% of its vCPUs) is held for 10 minutes, and its memory must not keep growing;
- the maximum, the highest rate stored in full, is held for 10 minutes too. A hold that fails is run again once before it counts.
The results report the hold, not the one-minute step, because a one-minute peak overstates what a machine sustains: the historian's background work (compacting what it has written) is paid for minutes later. On a dry run before this session, a one-minute peak was higher than the rate the same machine then held.
Pipelines. Each class climbs 30 to 50% a step from a fraction of its expected rate, each step held for 40 seconds (45 for the in-order runs), until a step breaks a capacity rule; the rate is then refined between the last passing and the first failing step. A failing step is run again once before it counts. A step passes when:
- at least 95% of the offered runs executed, and no more than 0.1% failed,
- the runs waited less than 1 s in the execution queue (p99), and the derived value arrived less than 1 s after the device's publish (p99),
- MaestroHub's CPU stayed under 90% of its vCPUs,
- the load generator stayed under 85% of its own cores.
After each class, the derived topics of 40 sampled source topics were read back by value: every value a device had published had to be there.
What limited each result is classified from counters read at the limiting step: MaestroHub's container CPU and CPU throttling; CPU, disk and memory pressure; disk MB/s and IOPS against what fio measured on the disk; network against what iperf3 measured; the historian's write stalls; and the load generator's own CPU. The classes are:
| Class | Meaning |
|---|---|
| CPU | MaestroHub's vCPUs were full. A bigger machine stores more. |
| Execution slots | All 256 execution slots of the priority lane were busy while the CPU had room. A setting, see Settings. |
| Software limit | Nothing was saturated and the CPU had room: a limit in MaestroHub's code, which we treat as a defect to fix. |
| Undetermined | Nothing was saturated, and the CPU was between the software-limit bound and saturation. |
Disk, network, memory and the load generator never limited a result in this session.
Ingest results
Measured 16 October 2026, MaestroHub 3.0. CPU is MaestroHub's mean, in busy logical cores.
| Machine | Result | Stored values/s | Publish to stored p99 | CPU | Memory (max) |
|---|---|---|---|---|---|
c3-highcpu-4 | Guaranteed, 10-minute hold | 29,852 | 56.1 ms | 3.11 of 4 (78%) | 681 MB |
| Maximum, 10-minute hold | 39,865 | 160.8 ms | 3.84 of 4 | 755 MB | |
c3-highcpu-8 | Guaranteed, 10-minute hold | 59,905 | 34.3 ms | 5.87 of 8 (73%) | 772 MB |
| Maximum, 10-minute hold | 89,248 | 161.8 ms | 7.87 of 8 | 994 MB | |
c3-highcpu-22 | Guaranteed, 10-minute hold | 164,376 | 26.2 ms | 16.1 of 22 (73%) | 893 MB |
| Maximum, 10-minute hold | 198,982 | 53.5 ms | 19.39 of 22 | 985 MB |
- Every hold stored what was offered (offered and stored differ by at most 4 values/s), and every read-back found every acknowledged value. No value was refused and the broker dropped none.
- The 8 vCPU maximum was held on the second attempt: the first 10-minute hold at 90,000 values/s stored just under 99% of the target, the second stored it in full.
- CPU per value: at the guaranteed holds MaestroHub stored 9,599, 10,205 and 10,210 values/s per busy logical core, about 0.1 ms of CPU per value on this machine type.
- Scaling: the guaranteed rate doubled from 4 to 8 vCPU, and grew 5.5 times from 4 to 22 vCPU, for 5.5 times the cores: about 7,500 values/s per vCPU on every size.
- Memory during the 10-minute holds grew by 1.6, 1.8 and 2.5 MB per minute at the guaranteed rates, and by 5.7, 3.5 and 3.7 MB per minute at the maximums. Longer holds were not run.
- Disk: at the guaranteed holds MaestroHub wrote 5.9, 10.0 and 32.8 MB/s on average (39, 56 and 151 write operations per second). The highest 5-second sample of data-disk writes in the whole session, pipeline tests and topic creation included, was 92, 118 and 161 MB/s.
- Network: at the guaranteed holds MaestroHub received 4.4, 8.8 and 24.7 MB/s.
- First contact: at the start of every ingest run, 10,000 new tags each sent their first value at once, and MaestroHub created their topics in 13 to 18 seconds. Because a value is acknowledged only once it is stored, 3,000 to 5,000 of those first values were acknowledged later than the load generator's 10-second timeout; every one of them was stored.
The curves, step by step
Each step held one minute. The guaranteed step is the last one meeting the rule; the steps after it show where the rule broke.
c3-highcpu-4: 4 vCPU = 2 physical × 2 threads
| Offered values/s | Stored values/s | Publish to stored p99 | CPU (cores of 4) | |
|---|---|---|---|---|
| 5,000 | 4,992 | 3.4 ms | 0.90 | |
| 10,000 | 9,977 | 4.9 ms | 1.41 | |
| 20,000 | 19,940 | 26.9 ms | 2.36 | |
| 30,000 | 29,842 | 51.2 ms | 3.08 | guaranteed |
| 40,000 | 39,824 | 159.7 ms | 3.85 | maximum stored in full; p99 over 100 ms, CPU over 80% |
| 50,000 | 49,080 | 337.9 ms | 3.95 | devices slowed: 98% of the target sent, confirmed on a second run |
c3-highcpu-8: 8 vCPU = 4 physical × 2 threads
| Offered values/s | Stored values/s | Publish to stored p99 | CPU (cores of 8) | |
|---|---|---|---|---|
| 5,000 | 4,997 | 1.8 ms | 0.93 | |
| 10,000 | 9,994 | 1.8 ms | 1.42 | |
| 20,000 | 19,988 | 3.3 ms | 2.42 | |
| 30,000 | 30,000 | 7.1 ms | 3.17 | |
| 40,000 | 39,972 | 16.3 ms | 4.11 | |
| 50,000 | 49,905 | 25.2 ms | 4.85 | |
| 60,000 | 59,968 | 36.9 ms | 5.90 | guaranteed |
| 75,000 | 74,947 | 47.4 ms | 7.02 | CPU over 80% |
| 90,000 | 89,615 | 123.9 ms | 7.86 | maximum stored in full; p99 over 100 ms |
| 110,000 | 102,369 | 239.6 ms | 7.91 | devices slowed: 93% of the target sent |
c3-highcpu-22: 22 vCPU = 11 physical × 2 threads
| Offered values/s | Stored values/s | Publish to stored p99 | CPU (cores of 22) | |
|---|---|---|---|---|
| 5,000 | 4,997 | 1.4 ms | 0.93 | |
| 10,000 | 9,993 | 1.3 ms | 1.53 | |
| 20,000 | 19,977 | 1.2 ms | 2.61 | |
| 30,000 | 29,944 | 1.5 ms | 3.39 | |
| 40,000 | 39,906 | 2.3 ms | 4.34 | |
| 50,000 | 49,900 | 3.1 ms | 5.18 | |
| 60,000 | 59,864 | 3.9 ms | 6.22 | |
| 75,000 | 74,835 | 5.9 ms | 7.56 | |
| 90,000 | 89,740 | 7.5 ms | 8.87 | |
| 110,000 | 109,508 | 12.9 ms | 10.99 | |
| 135,000 | 134,458 | 17.5 ms | 13.24 | |
| 165,000 | 164,411 | 26.6 ms | 16.35 | guaranteed |
| 200,000 | 198,952 | 53.5 ms | 19.79 | maximum stored in full; CPU over 80% |
| 240,000 | 222,612 | 110.6 ms | 21.14 | devices slowed: 93% of the target sent |
On every machine, latency stays low while the CPU has room and climbs once the vCPUs are close to full.
Pipeline results
Measured 16 October 2026, MaestroHub 3.0. Runs/s is the rate of the highest step that passed every capacity rule, with no run lost in the read-back; where the run log records a lower executed rate for that step, the executed rate is shown. p99 is from the device's publish to the derived value. CPU per run is MaestroHub's CPU divided by the runs, so it includes the device value that triggered the run and the derived value it stored.
c3-highcpu-4: 4 vCPU = 2 physical × 2 threads
| Class | Runs/s | p99 | CPU (cores) | CPU per run | Limited by |
|---|---|---|---|---|---|
| Pass-through | 4,400 | 124 ms | 3.31 | 0.75 ms | CPU |
| Pass-through, second run | 4,400 | 158 ms | 3.44 | 0.78 ms | CPU |
| 20 ms of script work | 86 | 136 ms | 2.80 | 32.7 ms | CPU |
| Waits 20 ms | 4,400 | 140 ms | 3.23 | 0.74 ms | CPU |
| Pass-through, in order | 2,900 | 135 ms | 3.48 | 1.20 ms | CPU |
c3-highcpu-8: 8 vCPU = 4 physical × 2 threads
| Class | Runs/s | p99 | CPU (cores) | CPU per run | Limited by |
|---|---|---|---|---|---|
| Pass-through | 8,400 | 82 ms | 6.42 | 0.77 ms | CPU |
| Pass-through, second run | 10,000 (9,977 executed) | 130 ms | 7.09 | 0.71 ms | CPU |
| 20 ms of script work | 170 | 82 ms | 5.54 | 32.6 ms | CPU |
| Waits 20 ms | 8,000 | 68 ms | 5.83 | 0.73 ms | CPU |
| Pass-through, in order | 4,500 | 70 ms | 6.03 | 1.35 ms | CPU |
c3-highcpu-22: 22 vCPU = 11 physical × 2 threads
| Class | Runs/s | p99 | CPU (cores) | CPU per run | Limited by |
|---|---|---|---|---|---|
| Pass-through | 18,000 | 47 ms | 15.06 | 0.84 ms | undetermined |
| Pass-through, second run | 18,000 | 70 ms | 16.55 | 0.92 ms | undetermined |
| 20 ms of script work | 480 | 59 ms | 16.62 | 34.7 ms | CPU |
| Waits 20 ms | 10,000 | 38 ms | 8.89 | 0.89 ms | execution slots |
| Pass-through, in order | 6,000 | 26 ms | 7.75 | 1.30 ms | software limit |
- Pass-through stopped at 82 to 84% CPU on the default 256 execution slots per lane, in both runs.
- Waits 20 ms is bounded by the execution slots. Each run holds its slot for about 22 ms, so 256 slots carry about 11,000 runs per second however many cores there are. At the limiting step all 256 slots were busy with the CPU at about half of the 22 cores.
- In order stopped with the CPU at 10.4 of 22 cores, a software limit.
Beside ingest
With device ingest at 65% of the machine's guaranteed rate running on other topics:
| Machine | Ingest beside | Pass-through | 20 ms of script work |
|---|---|---|---|
c3-highcpu-4 | 19,500 values/s | 1,700 runs/s, p99 202 ms, 3.59 of 4 cores | 65 runs/s, p99 217 ms, 3.52 of 4 cores |
c3-highcpu-8 | 39,000 values/s | 4,500 runs/s, p99 139 ms, 7.15 of 8 cores | 130 runs/s, p99 148 ms, 7.05 of 8 cores |
c3-highcpu-22 | 107,250 values/s | 8,300 runs/s, p99 47 ms, 17.1 of 22 cores | 360 runs/s, p99 81 ms, 19.58 of 22 cores |
Every result was limited by CPU, and the background devices' read-back found every acknowledged value.
Ingest and pipelines share the same cores. Once a workspace has a UNS trigger, every value stored in it is matched against the deployed triggers, including values no trigger subscribes to.
A fleet of pipelines
Each pipeline on a tag of its own, each run once a second, each size held for 90 seconds:
| Machine | Pipelines | Runs/s | p99 | CPU (cores) | Data disk writes |
|---|---|---|---|---|---|
c3-highcpu-4 | 1,000 | 997 | 23.8 ms | 1.04 | 27.9 MB/s, 2,687 operations/s |
| 2,000 | 1,998 | 96.3 ms | 2.47 | 73.8 MB/s, 7,534 operations/s | |
c3-highcpu-8 | 1,000 | 998 | 8.8 ms | 1.11 | 27.7 MB/s, 2,718 operations/s |
| 2,000 | 1,998 | 6.9 ms | 2.48 | 74.2 MB/s, 7,575 operations/s | |
c3-highcpu-22 | 1,000 | 998 | 6.8 ms | 1.12 | 27.9 MB/s, 2,711 operations/s |
| 2,000 | 1,999 | 3.8 ms | 2.57 | 74.4 MB/s, 7,584 operations/s |
- No run failed, and 16 sampled derived topics read back held every value their devices had sent. The fleet was verified by sample only.
- A deployed, idle pipeline used about 360 to 390 KB of memory.
- A pipeline that runs once a second gets a history row for every run (about 1.3 KB of run data at Standard), so the history's disk use grows with the fleet: at 2,000 pipelines the disk did about 7,500 write operations per second, three quarters of the 10,000 provisioned on the 4 vCPU machine.
Above the limit
Device ingest: devices slow down, nothing acknowledged is dropped
On the embedded broker, MaestroHub handles arriving values on a bounded set of workers. When every worker is busy, the next value is handled on the sending connection's own read loop, so the broker reads that device's connection more slowly. The device sees its acknowledgements (PUBACK) arrive later, because each one waits for its value to be stored; once it has as many messages in flight as its window allows, it keeps the rest in its own buffer. Nothing is queued without bound inside MaestroHub and nothing is dropped.
In the steps past the maximum the load generator could no longer offer the target rate (on the 4 vCPU machine, 49,080 values/s stored of 50,000 targeted), because its devices' in-flight windows were full. The read-back found every acknowledged value stored.
What this means for your devices:
- Use QoS 1. A QoS 1 client keeps every value until it is acknowledged. At QoS 0, a device that cannot send loses the value on the device.
- Give the device a buffer. A device that is slowed holds its values until the broker reads them. How long it can wait depends on its buffer, not on MaestroHub.
- Bound the in-flight window (MQTT 5 Receive Maximum, or the client's max-inflight setting). The embedded broker disconnects a client that sends more than
modules.uns.embeddedBrokerLimits.receiveMaximum(1,000) unacknowledged messages.
What an acknowledgement means
The embedded broker acknowledges a value (PUBACK) only once MaestroHub has stored it or refused it. A device that has its acknowledgement can forget the value; a device without one still holds it and sends it again after a reconnect. A value refused (for a permission, a rate limit or a value its topic cannot take) is answered with the reason, and an MQTT 5 device gets it in the PUBACK. See Acknowledged once decided.
"Stored" means written to the historian. With the built-in historian's default, a written value reaches the disk within 100 ms: a clean stop or restart loses nothing, but a crash (a killed process, a power loss) can lose the last ~100 ms of acknowledged values. Where a crash must lose nothing, turn on historian.pebble.syncWrites, at some cost in throughput (not measured in this session), or use the TimescaleDB historian. Every figure on this page was measured with the default.
Live subscribers are a separate path: for a subscriber that reads or acknowledges more slowly than values arrive, messages wait in its queue (maxSubscriberQueue, 10,000 by default); once the queue is full the oldest queued message is dropped for each new one, so the subscriber keeps receiving the newest values. Storage is not affected. See Capacity limits.
Pipelines: runs wait in the execution queue
A trigger that fires faster than the engine runs puts its runs in the execution queue, a stream in MaestroHub's embedded message bus (on the data disk in this test). They wait there and run later, in priority order. The queue is bounded: by default modules.pipelineEngine.executionQueue.maxBytes is 64 MiB and maxAge 1 hour. When it is full, the oldest waiting runs are dropped to admit new ones and counted, and the Overview says so (see below). Set executionQueue.discardNewOnFull: true to refuse new runs instead. In the pipeline tests no run was dropped.
What an operator sees
On the Overview:
- A replica is starved of CPU when runnable work waits 1 s or more (p99) for a core, or waited for a core in 50% or more of the last 10 seconds. The issue names the replica, the delay, any CPU-quota throttling, and the three pipelines of your workspace that used the most CPU in the last 15 minutes. It turns critical at a 60 s delay.
- Executions were stranded when runs reached the engine's broker connection but not the engine. They are not lost: the broker delivers them again after
executionQueue.scheduler.ackWait(5 minutes by default), so they run late. The issue's remedy names a CPU-starved engine as the first thing to check. - Queued executions were shed when the execution queue hit its size or age cap.
Metrics to watch (Prometheus names):
| Metric | What it tells you |
|---|---|
maestrohub_runtime_sched_latency_p99_seconds | How long runnable work waits for a core. Grows when the CPU is full. |
maestrohub_runtime_cpu_pressure_percent | Share of the last 10 seconds in which work waited for a core. |
maestrohub_pipeline_cpu_seconds_total | CPU used per pipeline. |
maestrohub_uns_ingress_accepted_total, maestrohub_uns_ingress_rejected_total | Values admitted, and values refused by reason (for example rate_limited). |
maestrohub_uns_storage_write_duration_seconds | Historian write time. |
maestrohub_uns_pebble_write_stall_seconds_total | Time the historian held writes back. |
maestrohub_engine_queue_wait_seconds | How long runs waited in the execution queue, per priority. |
maestrohub_engine_queue_consumer_pending, maestrohub_engine_queue_consumer_inflight | Runs waiting and runs running, per priority lane. |
maestrohub_engine_execution_queue_shed_total | Waiting runs dropped at the queue's cap. |
maestrohub_engine_queue_deliveries_stranded_total | Deliveries that will run late, after ackWait. |
maestrohub_uns_broker_messages_dropped_total | Messages dropped for slow live subscribers. |
Settings that change the numbers
Each key is a config.yaml key (or the MAESTROHUB_ environment variable of the same path) unless it is named as a pipeline setting. Defaults are those of MaestroHub 3.0.
| Setting | Default | What it does |
|---|---|---|
modules.uns.ingress.perClientRatePerSec | 2000 | Values per second one device or gateway may write. Above it, values are refused as rate_limited. 0 turns the limit off. Off in the test. |
modules.uns.ingress.perWorkspaceRatePerSec | 20000 | Values per second one workspace may write through the MQTT write endpoint, all devices together. Raise it to use what a larger machine can store: at the default, a workspace is capped at 20,000 values/s on every machine size. 0 turns the limit off. Off in the test. |
modules.uns.embeddedBrokerLimits.maxClients | 5000 | MQTT clients connected at once. The test used 1,000 devices. |
modules.uns.embeddedBrokerLimits.receiveMaximum | 1000 | Unacknowledged QoS 1 messages one client may send. |
modules.pipelineEngine.executionQueue.scheduler.maxAckPending | 256 | Execution slots per priority lane: how many runs of a lane may run at once. Three lanes, so at most 768 runs at once. The limit is held by the queue, so it is shared by every engine replica. A running pipeline holds one QoS 1 publish on its system.uns.publish node, so keep 3 × this value at or below receiveMaximum. Pipelines that mostly wait (an HTTP call, a slow read) reach this limit before the CPU: about 256 ÷ the run time runs per second per lane. |
| Pipeline setting Priority | Normal | High, Normal or Low. Each priority has its own lane and its own slots. The engine takes from the highest lane first; a lower lane passed over 100 times in a row (executionQueue.scheduler.starvationThreshold) gets one run taken from it. |
| Pipeline setting Run order | Parallel | One at a time, in arrival order keeps a pipeline's runs in sequence. With 20 pipelines it ran 33 to 66% of the parallel rate, at 1.4 to 1.9 times the CPU per run. |
| Pipeline setting Execution History Detail | Standard | Minimal stores the status only, Standard adds each node's status and timing, Full adds every node's data. At every level, a pipeline started by a trigger stores at most one successful run per second; failed runs and runs a person started are always stored. |
modules.pipelineEngine.executionRepository.sqlite.retention | 24h | How long the execution history is kept. |
modules.pipelineEngine.executionQueue.maxBytes, maxAge, discardNewOnFull | 64 MiB, 1 h, false | The execution queue's cap, and what happens when it is full (see Above the limit). |
| History per topic | 1 day | Each topic keeps 1, 7, 14 or 32 days of history. It sets the disk space history takes; the test kept 32 days. History you need for longer belongs in your own system of record. See How history retention works. |
modules.uns.liveMirror.qos | 1 | QoS of the live re-publish of every stored value. The test ran at 1. |
Not measured in this session: Minimal and Full execution history, High and Low priority, QoS 0 for the live re-publish, and an execution queue kept in memory.
Sizing a deployment
-
Add up the values per second at your busiest minute: devices × tags ÷ the interval at which each tag is sent.
-
Pick the machine by its guaranteed rate, and leave room to grow. The guaranteed rate already keeps 20% or more of the CPU free; do not plan on the maximum.
-
Add the pipelines' CPU. Ingest and pipelines share the same cores. On this machine type, measured:
- about 0.1 ms of CPU per stored value (10,000 values/s per busy logical core),
- about 0.8 ms per pass-through run, counting the run, the value that triggered it and the derived value it stored,
- about 35 ms per run of 20 ms of script work.
Keep the total at or under three quarters of the vCPUs, where the guaranteed holds ran (73 to 78%).
-
Check the pipelines that wait. A pipeline that spends most of its run waiting (an HTTP call, a slow machine read) is bounded by its lane's 256 execution slots, not by the CPU: about 256 ÷ its run time per second.
-
Check the defaults. Raise
modules.uns.ingress.perWorkspaceRatePerSecabove your devices' rate, andmodules.uns.embeddedBrokerLimits.maxClientsabove your device count. -
Confirm on your hardware. A different CPU, disk or cloud gives different figures. We can run this test with you before you commit to a machine.
The test sent one value per message. The MQTT write endpoint stores one value per message (a JSON object without a value key is stored as one structured value), so the ingest rates above are messages per second as well.
A worked example
A plant has 400 devices with 25 tags each, every tag sent once a second, plus 300 pass-through pipelines that each run once a second and 10 pipelines that each run 20 ms of script once a second.
| Load | Per second | CPU per item | Busy cores |
|---|---|---|---|
| Device values | 10,000 | 0.1 ms | 1.0 |
| Pass-through runs | 300 | 0.8 ms | 0.24 |
| Script runs | 10 | 35 ms | 0.35 |
| Total | about 1.6 |
- A 4 vCPU machine of this type has 3 cores under the three-quarter line. 1.6 cores fits, with room for the load to grow by about 85%.
- 10,000 values/s fits under the default workspace limit of 20,000.
- If the plant doubles to 20,000 values/s and 600 pass-through runs/s, the estimate is about 2.8 cores, close to the line: move to an 8 vCPU machine, and raise the workspace limit above its default of 20,000.
The estimate adds measured per-item costs; it is not itself a measurement. For the two mixes measured beside ingest on the 4 vCPU machine it gave 3.3 and 4.2 cores where the machine used 3.59 and 3.52, within about 20% either way. Keep the three-quarter line as your margin, and test your own mix.
What the test does not cover
- Enterprise: Kubernetes, several nodes, replicas, external PostgreSQL, TimescaleDB or EMQX. Only Tier 1 on one machine was measured.
- Other machines: ARM processors, other Intel or AMD generations, other clouds, bare metal, Windows.
- Other disks: only Google Cloud Hyperdisk Balanced with provisioned IOPS and throughput. Slower disks were not measured.
- Large payloads: every value was a single number. Strings, objects and arrays were not measured.
- Many live subscribers on the data: no MQTT subscriber read the stored values during the ingest test.
- QoS 0, MQTT over TLS and WebSocket clients.
- More than 1,000 devices, and connection storms.
- Reading history under load: no queries or dashboards ran during the tests.
- Long runs: the longest hold was 10 minutes; pipeline steps lasted 40 to 90 seconds. Multi-hour and multi-day runs are not in these figures.
syncWriteson: every figure is with the historian's default. Withhistorian.pebble.syncWriteson, a crash loses nothing acknowledged, at a throughput cost not measured here.- Crashes: no run killed MaestroHub. What a crash loses is tested separately (
perf/bench/killtest.shin the MaestroHub repository). - Pipelines with real work: connector reads and writes, HTTP calls, databases, and triggers other than UNS triggers.
Measured on 16 October 2026, MaestroHub 3.0, Google Cloud c3-highcpu-4, c3-highcpu-8 and c3-highcpu-22 in europe-west4-b. We run the test again before every release and update this page with the release it was measured on.