Connection Timeouts
MaestroHub composes every operation's effective deadline from six layers,
applied in order from broadest to most specific. Each layer is shorten-only:
whichever bound is tightest wins. Once composed, the deadline flows to every
protocol client via ctx.Deadline() — protocol clients trust it and do not
re-read timeout configuration.
The six layers
Layer 1 · Pipeline pipeline.Timeout 30 min (safety ceiling; settable up to 24 h)
Layer 2 · Node node.Settings.Basic.Timeout unset by default
Layer 3 · Override node.params.timeoutOverride unset (per-invocation)
Layer 4 · Function function.RequestTimeout unset by default
Layer 5 · Connection connection.RequestTimeout unset by default
Layer 6 · Connect connection.ConnectTimeout 10 sec (dial ceiling)
Layers 1 and 6 apply defaults even when the field is unset (safety ceilings). Layers 2–5 drop out when unset — a higher layer's bound wins by default.
Setting a pipeline's run timeout
Layer 1 bounds a whole run of the pipeline — every node, start to finish.
A pipeline that sets nothing may run for 30 minutes. To let a pipeline
run longer (or stop it sooner), set its Run timeout: open the pipeline's
settings in the designer, Advanced → Run timeout, and enter a duration
such as 45m, 6h or 24h. Anything from 1s to 24h is accepted; leave
the field empty to go back to the default.
Through the API, send timeout as a Go duration string on create or
update (PUT /engine/pipelines/:id, or update_meta in
PATCH /engine/pipelines/:id/changes):
{ "timeout": "6h" }
"" clears it. A value outside 1s–24h is refused with a 400 that names
the limit. Every pipeline response carries timeout (absent when unset) and
effectiveTimeout — the limit a run actually gets.
A run that reaches its timeout is stopped and fails with
pipeline execution timeout exceeded (6h0m0s). Each node inside the run is
still bounded by its own timeout (Layer 2), so a long run is made of nodes
that each fit theirs.
Which connectors enforce Layer 5?
Every connector honors Layers 1–4 (composed at the engine layer). Layer 5
(per-request connection timeout) applies when the connection has
RequestTimeout set explicitly. Whether the timeout also translates to a
server-side abort depends on the underlying protocol. Below is the honest
per-connector table.
Full server-side enforcement
The server aborts the operation at the deadline and stops consuming resources. This is the case that also stops billing on paid backends.
| Connector | Mechanism |
|---|---|
| PostgreSQL | SET statement_timeout per query |
| QuestDB | SET statement_timeout (Postgres wire) |
| MySQL | SET SESSION MAX_EXECUTION_TIME |
| ClickHouse | SET max_execution_time |
| Snowflake | ALTER SESSION SET STATEMENT_TIMEOUT_IN_SECONDS |
| Databricks SQL | SET statement_timeout |
| BigQuery | Query.JobTimeout (SDK) |
| Athena | StopQueryExecution on ctx expiry (reactive) |
| MongoDB | Find/Aggregate.SetMaxTime (SDK) |
| Elasticsearch | Search.WithTimeout → ?timeout= |
| Prometheus | promv1.WithTimeout → ?timeout= |
Client-side only
The client cancels its wait at the deadline. The server may keep running the operation until it independently notices the socket dropped. Typical of protocols with no server-side deadline hint mechanism.
| Connector | Notes |
|---|---|
| AWS Lambda | Function's own configured timeout controls execution; MaestroHub's ctx only bounds the client's wait. You still pay for the full function duration. |
| REST / SOAP | No standard timeout header. Some services accept X-Timeout or ?timeout= as custom headers/params — configure per-connection if the downstream service supports it. |
| MSSQL | No native server-side statement timeout. Resource Governor is DBA-level configuration. |
| Oracle | go-ora driver sends CANCEL on ctx expiry — the server usually aborts promptly. Effectively similar to server-side for most operations. |
| ODBC | WebSocket adapter architecture. Server-side timeout would need to live in the adapter binary. |
| InfluxDB | Client-side HTTP request timeout; InfluxQL has no per-query server timeout. Flux queries have some capability but not wired. |
| Redis | Commands are microsecond-scale; ctx cancel is sufficient. |
| Kafka producer | delivery.timeout.ms at connection config; per-message tuning not exposed. |
| RabbitMQ, NATS, MQTT | Publish is fire-and-ack; no server-side "abort my publish" mechanism. Broker either commits or doesn't. |
| SMTP | Short session; ctx cancel closes the connection. |
| HTTP notifications (Slack, MS Teams) | Short HTTP request; ctx cancel is sufficient. |
Streaming / socket close is sufficient
Operations either stream (HTTP body) or hit a fast sync response — closing the socket immediately stops server-side work. No server-side abort mechanism is needed.
| Category | Connectors |
|---|---|
| Object storage | S3, Azure Blob, Google Cloud Storage, OneLake, Azure IoT Hub |
| File | Local File, FTP, SMB |
| Cache | Redis |
| Cloud messaging | Kinesis, SNS, SQS, Google Pub/Sub |
IoT / industrial-control
Sync TCP request-response protocols. Socket close on ctx expiry is instant server-side abort. Server holds no state after replying.
Modbus, OPC UA, OPC DA, S7, EtherNet/IP, BACnet, TwinCAT, CANopen, Melsec, Omron, MTConnect, Ignition, SiteWise, LoRaWAN, Sparkplug B.
Setting a connection's requestTimeout
A connection's Request Timeout setting (requestTimeout in its config) is
the single source of truth for Layer 5. MaestroHub applies it to every
operation on the connection. It is read in the format the connector declares,
so you never have to think about units.
In connection config JSON:
{
"requestTimeout": "30s"
}
Accepted formats: any Go duration string — "30s", "5m", "1h30m",
"500ms". A plain number saved by an older version is read as seconds.
Precedence:
- The connection's
requestTimeout, if the connection has one - Otherwise Layer 5 drops out; the effective deadline comes from the other layers
A connection created in the UI saves the form's Request Timeout value. A
connection created through the API without requestTimeout has no Layer 5:
a connector's default value is only a suggestion in the form, never an
implicit bound.
When to set it:
- You have specific per-connection latency expectations (e.g., "no Snowflake query should exceed 5 min on this warehouse")
- You want a bound tighter than the pipeline's default 30-min ceiling without setting it on every node
When to leave it unset:
- The 30-min pipeline ceiling is fine for your workload
- You'd rather set the bound at the pipeline or node level so it's explicit per-run
Setting a connection's connectTimeout
Layer 6 bounds how long the connector spends opening the connection (TCP
dial + TLS handshake + auth exchange). Applied once at Connect(), not
per request. Independent from Layers 1–5.
Default: the connector's own Connect Timeout default, shown in its form.
In connection config JSON:
{
"connectTimeout": "15s"
}
When to raise it:
- Slow authentication mechanisms (Kerberos, some SSO flows)
- Distant cloud regions where TLS handshake dominates
- Backhaul networks with high latency
Layer composition examples
Case 1: Pipeline sets a tight cap, everything below is unset.
Pipeline: 60s
Node: unset
Function: unset
Connection: unset
Effective deadline: 60s (Layer 1 wins by construction).
Case 2: User sets a Snowflake connection cap of 5 min but the pipeline is 30 min.
Pipeline: 30 min
Node: unset
Function: unset
Connection: 5 min
Effective deadline: 5 min (Layer 5 tighter than Layer 1).
Case 3: A node explicitly overrides to 90s for a slow batch operation.
Pipeline: 30 min
Node: 90s
Function: unset
Connection: 5 min
Effective deadline: 90s (Layer 2 tightest).
Case 4: Nothing is set anywhere.
Pipeline: unset → 30 min default
Node: unset
Function: unset
Connection: unset
Effective deadline: 30 min (Layer 1 safety ceiling).
Common pitfalls
A function's timeout is its own requestTimeout setting. Legacy
timeoutMs / timeoutSeconds keys on older functions are still honored.
When a function and its connection both set a timeout, the tighter one wins.
AWS Lambda's execution timeout is not observable via ctx. Lambda
functions run to their configured maximum regardless of what MaestroHub's
ctx says. If your Lambda is configured for 15 minutes and you set
a connection requestTimeout of 30s, MaestroHub returns after 30s but you
pay for the full Lambda duration.
Long-poll operations must fit within the composed budget. After the
timeout-hierarchy refactor, SQS's Receive with waitTimeSeconds: 20
against a connection requestTimeout of 15s cuts short at 15s. Set the
pipeline/connection timeout to comfortably exceed the long-poll window
plus a few seconds of headroom.