Prometheus Integration Guide

Prometheus is the dominant open-source observability metrics store. MaestroHub's Prometheus connector reads metrics with PromQL over the HTTP query API and writes samples with the remote_write protocol, so a pipeline can pull SRE metrics into plant logic — and push plant telemetry back out into the same metrics stack the rest of your infrastructure already uses.

Overview

The Prometheus connector delivers:

• Instant PromQL queries (/api/v1/query) — evaluate an expression at a single point in time
• Range PromQL queries (/api/v1/query_range) — evaluate an expression across a start/end window at a fixed step
• remote_write (snappy-compressed protobuf) to any Prometheus-compatible sink
• Metric name discovery via /api/v1/label/__name__/values
• Bearer token, basic auth, or unauthenticated access
• Mutual TLS with optional CA and client certificate
• Configurable HTTP request timeout on the connection and per operation

Which servers this works against

The read path targets any server that implements the Prometheus HTTP API. The write path targets any server that accepts remote_write — Prometheus itself (with remote_write receiving enabled), Grafana Mimir, Cortex, Thanos Receive, and VictoriaMetrics. Non-Prometheus sinks usually expose remote_write on a different path; set Remote-Write Path accordingly.

Connection Configuration

Creating a Prometheus Connection

Navigate to Connections → New Connection → Prometheus and fill in these details.

Prometheus Connection Creation Fields

1. Profile Information

Field	Default	Description
Profile Name	-	A descriptive name for this connection profile (required, max 100 characters)
Description	-	Optional description for this Prometheus connection

2. Connection

Field	Type	Default	Validation	Description
Server URL	String	-	Required	Prometheus HTTP API base URL, e.g. http://localhost:9090. A trailing slash is trimmed.
HTTP Request Timeout	Duration	30s	1s–1h	Timeout applied to every read and write request
Remote-Write Path	String	/api/v1/write	-	Path appended to the Server URL for remote_write. Override for Cortex/Mimir/VictoriaMetrics compatibility (e.g. /api/v1/push). A leading / is added if you omit it.

3. Authentication

Field	Type	Default	Description
Authentication Mode	Select	none	How to authenticate every HTTP request — none, basic, or bearer
Username	String	-	Username for Basic authentication (used when Authentication Mode is basic)
Password	Password	-	Password for Basic authentication. Stored encrypted; leave empty on edit to keep the stored value.
Bearer Token	Password	-	Token for Bearer authentication (used when Authentication Mode is bearer). Stored encrypted.

The auth header is applied to every request the connector makes — the reachability probe, both query paths, metric discovery, and remote_write.

4. TLS

Field	Type	Default	Description
Skip TLS Verification	Boolean	false	Skip TLS certificate verification (not recommended for production)
CA Certificate (PEM)	String	-	Custom CA certificate for verifying the server's certificate
Client Certificate (PEM)	String	-	Client certificate for mutual TLS authentication
Client Private Key (PEM)	Password	-	Private key for the client certificate. Stored encrypted.

Skip TLS Verification is for development only

Turning it on disables certificate validation for every request on this connection, including remote_write. Use a CA certificate instead when the server presents a private or self-signed certificate.

5. Connection Labels

Field	Default	Description
Labels	-	Key-value pairs to categorize and organize this Prometheus connection (max 10 labels)

Validation Rules

These cross-field rules are checked when the connection is saved:

Rule	Behaviour when violated
Authentication Mode must be none, basic, or bearer	invalid authMode: <value>
basic mode needs credentials	Rejected when Username and Password are both empty: basic auth requires username and password
bearer mode needs a token	Rejected when Bearer Token is empty: bearer auth requires bearerToken
Client certificate and key travel together	Rejected when only one is set: clientCertificate and clientKey must be provided together

How the connection is verified

Connecting calls /api/v1/status/runtimeinfo — a small authenticated request that fails fast when the URL, the credentials, or TLS are wrong, rather than surfacing later as a failed query. The health check uses the same endpoint and reports its round-trip latency. Prometheus connections are stateless HTTP and are shared across MaestroHub replicas rather than pinned to one instance, with up to 16 concurrent in-flight operations per connection.

Function Builder

Creating Prometheus Functions

Once the connection is saved:

1. Go to the connection's Functions tab and start a new function
2. Choose Instant Query, Range Query, Write Samples, or List Metrics
3. Configure the operation's fields and template parameters

All four functions are testable from the function editor before you wire them into a pipeline.

Instant Query

Purpose: Evaluate a PromQL expression at a single point in time via /api/v1/query. Use it for dashboards, spot checks, and alerting expressions that don't need a time range.

Field	Type	Required	Default	Validation	Description
PromQL Query	String	Yes	-	-	PromQL expression to evaluate. Supports ((parameter)) syntax.
Evaluation Time	String	No	server time	-	RFC 3339 timestamp or Unix seconds. Supports ((parameter)) syntax.
Query Timeout	Duration	No	30m	1s–1h	Per-query execution timeout

Examples

up

sum by (job) (rate(http_requests_total[5m]))

Evaluation Time also accepts the literal now (case-insensitive) and a relative Go duration such as -1h, which resolves against the current time. Leaving it empty lets the server pick the evaluation instant.

Use Cases: current values for a KPI card, alert conditions, single-point lookups before a decision node

Range Query

Purpose: Evaluate a PromQL expression across a start/end range at a fixed step via /api/v1/query_range. This is the shape used for graphs, backfills, and downsampling.

Field	Type	Required	Default	Validation	Description
PromQL Query	String	Yes	-	-	PromQL expression to evaluate over the range. Supports ((parameter)) syntax.
Start	String	Yes	-	-	Range start — RFC 3339 timestamp, Unix seconds, or a relative offset like -1h. Supports ((parameter)) syntax.
End	String	No	now	-	Range end. Defaults to now if omitted. Supports ((parameter)) syntax.
Step	String	Yes	-	Must be > 0	Resolution step, e.g. 15s, 1m, 5m. A bare number is read as seconds. Supports ((parameter)) syntax.
Query Timeout	Duration	No	30m	1s–1h	Per-query execution timeout

Example

rate(node_cpu_seconds_total[5m])

Start is required and must parse to a real timestamp — an empty or unparseable value fails the call with start is required and must be a valid timestamp rather than silently defaulting.

The timeout reaches the server

Instant and Range queries forward the composed deadline to Prometheus as the ?timeout= query parameter, so an expensive expression is aborted by the query engine rather than left running after MaestroHub has stopped waiting. The deadline itself is composed from several shorten-only layers — pipeline, node, override, function, connection — and the tightest one wins; see Connection Timeouts. Metric discovery does not forward a ?timeout=, because that endpoint ignores it; it relies on cancellation instead.

Use Cases: trend charts, windowed shift reports, backfilling a historian from a metrics store

Write Samples

Purpose: Push one or more time-series samples using the remote_write protocol (snappy-compressed protobuf) to the connection's Server URL + Remote-Write Path.

Field	Type	Required	Default	Validation	Description
Default Metric Name	String	No	-	-	Default __name__ label applied when a sample does not carry its own. Supports ((parameter)) syntax.
Default Labels	Object	No	-	-	Labels merged into every sample in the batch. Supports ((parameter)) syntax.
Samples	Any	Yes	-	Non-empty array	Array of samples: [{value, timestamp?, labels?}, ...]. Supports ((parameter)) syntax.
Timeout	Duration	No	30m	1s–1h	Bound on this single operation

Sample shape

Every item in Samples is one object:

Property	Required	Description
value	Yes	The sample value. Numbers, numeric strings, and booleans (true → 1, false → 0) are all accepted.
timestamp	No	RFC 3339 (with nanoseconds) or an integer of Unix milliseconds. Defaults to the event time — see the note below.
metric	No	Overrides Default Metric Name for this sample
labels	No	Merged over Default Labels for this sample; per-sample keys win on collision

A metric name is mandatory on every sample, resolved from metric, from Default Metric Name, or from a labels.__name__ entry. A sample with none of those fails the call with data[i]: metric name is required (via 'metric' or labels.__name__).

Example configuration

{
  "metric": "plant_line_throughput_units_per_min",
  "labels": { "plant": "chicago", "line": "assembly-1" },
  "data": [
    { "value": "((throughput))", "labels": { "machine_id": "((machineId))" } },
    { "value": "((rejects))", "metric": "plant_line_rejects_total" }
  ]
}

Timestamps and store-and-forward

A sample with no timestamp is stamped with the event time — the runtime's produce time on a store-and-forward replay, or the current time on the synchronous path. A replayed sample therefore carries the moment it was produced, not the moment the buffer drained. Write Samples is store-and-forward eligible (sink, unordered, stale-tolerant, preferred batch size 500); because remote_write servers deduplicate by (labels, timestamp), attaching an explicit timestamp per sample gives a replay natural idempotency. See Store & Forward.

Non-2xx responses

A 2xx answer succeeds and reports the status code. Anything else fails the node, with the response body (first 4 KB) in the error message, and is classified per the remote_write spec:

Status	Classification	Effect
2xx	Success	samplesWritten and statusCode are returned
408, 425, 429	Transient	The store-and-forward buffer holds the batch and retries
Other 4xx	Permanent	Replaying the same malformed request would fail identically, so the batch is dead-lettered instead of held
5xx	Transient	Buffered and retried

Use Cases: publishing pipeline KPIs into the monitoring stack, bridging PLC or OPC UA readings into Prometheus, feeding Mimir/Thanos long-term storage from the plant floor

List Metrics

Purpose: List the metric names the server currently knows about, via /api/v1/label/__name__/values.

Field	Type	Required	Default	Validation	Description
Series Selector	String	No	-	-	Optional PromQL series selector (e.g. {job="node"}) to restrict the label scan. Supports ((parameter)) syntax.
Timeout	Duration	No	30m	1s–1h	Bound on this single operation

Use Cases: discovery flows, dashboard builders, validating that an expected metric exists before wiring an Instant or Range Query

Using Parameters

All four functions support the ((parameterName)) syntax in their templated fields — the query, the range bounds and step, the metric name, labels, sample data, and the series selector. Parameters are auto-detected: typing ((plant)) in a query adds plant to the function's parameter list, which then appears as an input on the pipeline node.

Configuration	Description	Example
Type	Validate incoming pipeline data	string, number, boolean, datetime, json, buffer
Required	Force parameter presence	Required / Optional
Default Value	Provide fallback values	'-1h', '5m', '{}'
Description	Document intent for other authors	"Range start resolved from the upstream node"

Understanding the Response

Instant Query and Range Query return the same shape: a resultType, a flat list of series — each with its labels and its points (a Unix-seconds timestamp and a value) — and the server's warnings, which is null when there are none.

How the four result types differ

resultType	Produced by	Series	Points per series	Labels
vector	An instant query returning an instant vector	One per matching series	Exactly one	The metric's labels, including __name__
matrix	A range query, or an instant query over a range selector	One per matching series	One per step in the range	The metric's labels, including __name__
scalar	A PromQL expression that evaluates to a single number, e.g. scalar(up) or 2 * 3	Exactly one	Exactly one	Empty
string	A PromQL string literal	Exactly one	Exactly one, whose value is a string rather than a number	Empty

Because scalar and string still arrive as a one-series, one-point list, a downstream expression can read every result type the same way — result.series[0].points[0].value.

NaN and Inf become 0

PromQL can evaluate to NaN or ±Inf — a division by zero, for instance. Those are not representable in JSON, so the connector emits 0 in their place, and a downstream node cannot tell that 0 apart from a real zero. If the distinction matters for your logic, write the expression so it cannot produce NaN or Inf in the first place.

List Metrics returns metrics (the names, as strings), metricCount, and warnings. Write Samples returns samplesWritten — one per item in Samples — and the statusCode the remote_write endpoint answered with.

The call's own facts ride under _metadata: query after an Instant or Range Query, step after a Range Query, and writeURL after Write Samples. List Metrics adds none.

For the full expression reference — every field, with the $node["Name"].result… path a pipeline uses to read it — see Prometheus Nodes.

Pipeline Integration

Use the functions you build here as nodes inside the Pipeline Designer. Drop in the Instant Query or Range Query node to pull metrics, the Write Samples node to publish, or the List Metrics node for discovery, then bind their parameters to upstream outputs or constants.

For the pipeline-side field reference and error-handling behaviour, see Prometheus Nodes. For patterns that combine Prometheus with SQL, MQTT, OPC UA, or other connector steps, see the Connector Nodes page.

Common Use Cases

Reading an SLO or KPI Metric Into Plant Logic

Use an Instant Query with an expression such as sum by (line) (rate(units_produced_total[15m])) to read the current production rate the monitoring stack already computes, then branch on it in the pipeline — raising an alert, throttling a downstream write, or annotating a batch record. This avoids recomputing in MaestroHub a number Prometheus already maintains.

Shift Reports From a Range Query

Use a Range Query with Start bound to ((shiftStart)), End to ((shiftEnd)), and a Step of 5m to pull one series per machine across a shift. Each series arrives with its labels and a full point list, ready to aggregate, write to a report, or archive to object storage.

Pushing Plant Telemetry Into Prometheus

Collect readings with an OPC UA, Modbus, or MQTT node, then feed them to Write Samples with Default Labels carrying the plant, line, and area, and one item per tag in Samples. Plant telemetry lands in the same metrics store as the IT infrastructure, so SRE dashboards and plant KPIs share one query language. With store-and-forward enabled, a monitoring outage buffers the batch instead of dropping it.

Long-Term Storage Without a Second Historian

Point Remote-Write Path at a Mimir, Cortex, Thanos Receive, or VictoriaMetrics ingest path and write directly to long-term storage. The read functions can then target the same cluster's query endpoint, so a pipeline writes and reads through one connector pair.

Discovery Before Wiring

Run List Metrics with a series selector such as {job="plc-exporter"} to confirm which metric names the server actually exposes, then wire the exact names into an Instant or Range Query. Cheaper than debugging an empty query result.

Troubleshooting

Symptom	Possible Cause	Solution
Save fails with Prometheus runtimeinfo check failed	Wrong Server URL, unreachable host, bad credentials, or a TLS problem	The connection probes /api/v1/status/runtimeinfo. Confirm the base URL (no API path — just scheme, host, port), that the host is reachable from the MaestroHub host, and that the Authentication Mode matches what the server expects.
Save fails with basic auth requires username and password	Authentication Mode is basic with both fields empty	Fill in Username and Password, or switch the mode to none.
Save fails with clientCertificate and clientKey must be provided together	Only one half of the mTLS pair is set	Provide both PEM values, or clear both.
TLS / certificate error at connect	Private or self-signed server certificate	Paste the issuing CA into CA Certificate (PEM). Use Skip TLS Verification only in development.
Query fails with invalid parameter "timeout"	The server rejected the forwarded deadline	The connector snaps the forwarded ?timeout= to a whole unit for exactly this reason; if it still appears, check for a proxy rewriting query parameters between MaestroHub and Prometheus.
Range Query fails with start is required and must be a valid timestamp	Start is empty or unparseable	Use RFC 3339, Unix seconds, or a relative offset like -1h. A templated parameter that resolves to an empty string hits this too.
Range Query fails with cannot parse "…" as step	Step is not a duration or a positive number	Use 15s, 1m, 5m, or a bare number of seconds. Zero and negative values are rejected.
Write fails with data must be a non-empty array of samples	The bound payload was empty, or is not a JSON array	Samples takes an array; bind it to an upstream array output or a JSON array literal.
Write fails with data[i]: metric name is required	Neither the sample nor the function defaults name a metric	Set Default Metric Name, or give the sample a metric or a labels.__name__.
Write fails with remote_write returned HTTP 404	The write path is wrong for this server	Prometheus uses /api/v1/write; Cortex and Mimir commonly use /api/v1/push. Set Remote-Write Path to match.
Write fails with remote_write returned HTTP 400 and the batch is dead-lettered	The server rejected the samples	4xx responses other than 408, 425, and 429 are permanent by design, since a replay would fail identically. The server's own response body is included in the error message — read it to see which sample it objected to, then fix the metric name, labels, or timestamps.
Write fails against a plain Prometheus with a 4xx and no store-and-forward retry	The target Prometheus is not accepting remote_write	Prometheus only receives remote_write when its receiver endpoint is enabled in the server's own configuration; check the server's docs for the release you run, or point the connection at a sink built to receive (Mimir, Cortex, Thanos Receive, VictoriaMetrics).
List Metrics returns fewer names than expected	The series selector is too narrow, or the metrics have aged out of the retention window	Drop the Series Selector to see the full set, then narrow it back down.
A value arrives as 0 when the metric looks non-zero	The expression evaluated to NaN or ±Inf	Those are coerced to 0 on the way out. Guard the expression in PromQL instead.