Connection Timeouts

MaestroHub composes every operation's effective deadline from six layers, applied in order from broadest to most specific. Each layer is shorten-only: whichever bound is tightest wins. Once composed, the deadline flows to every protocol client via ctx.Deadline() — protocol clients trust it and do not re-read timeout configuration.

The six layers

Layer 1 · Pipeline     pipeline.Timeout                              10 min (safety ceiling)
Layer 2 · Node         node.Settings.Basic.Timeout                    unset by default
Layer 3 · Override     node.params.timeoutOverride                    unset (per-invocation)
Layer 4 · Function     function.RequestTimeout                        unset by default
Layer 5 · Connection   connection.RequestTimeout                      unset by default
Layer 6 · Connect      connection.ConnectTimeout                      10 sec (dial ceiling)

Layers 1 and 6 apply defaults even when the field is unset (safety ceilings). Layers 2–5 drop out when unset — a higher layer's bound wins by default.

See Timeout Hierarchy Design for the architectural detail.

Which connectors enforce Layer 5?

Every connector honors Layers 1–4 (composed at the engine layer). Layer 5 (per-request connection timeout) applies when the connection has RequestTimeout set explicitly. Whether the timeout also translates to a server-side abort depends on the underlying protocol. Below is the honest per-connector table.

Full server-side enforcement

The server aborts the operation at the deadline and stops consuming resources. This is the case that also stops billing on paid backends.

Connector	Mechanism
PostgreSQL	SET statement_timeout per query
QuestDB	SET statement_timeout (Postgres wire)
MySQL	SET SESSION MAX_EXECUTION_TIME
ClickHouse	SET max_execution_time
Snowflake	ALTER SESSION SET STATEMENT_TIMEOUT_IN_SECONDS
Databricks SQL	SET statement_timeout
BigQuery	Query.JobTimeout (SDK)
Athena	StopQueryExecution on ctx expiry (reactive)
MongoDB	Find/Aggregate.SetMaxTime (SDK)
Elasticsearch	Search.WithTimeout → ?timeout=
Prometheus	promv1.WithTimeout → ?timeout=

Client-side only

The client cancels its wait at the deadline. The server may keep running the operation until it independently notices the socket dropped. Typical of protocols with no server-side deadline hint mechanism.

Connector	Notes
AWS Lambda	Function's own configured timeout controls execution; MaestroHub's ctx only bounds the client's wait. You still pay for the full function duration.
REST / SOAP	No standard timeout header. Some services accept X-Timeout or ?timeout= as custom headers/params — configure per-connection if the downstream service supports it.
MSSQL	No native server-side statement timeout. Resource Governor is DBA-level configuration.
Oracle	go-ora driver sends CANCEL on ctx expiry — the server usually aborts promptly. Effectively similar to server-side for most operations.
ODBC	WebSocket adapter architecture. Server-side timeout would need to live in the adapter binary.
InfluxDB	Client-side HTTP request timeout; InfluxQL has no per-query server timeout. Flux queries have some capability but not wired.
Redis	Commands are microsecond-scale; ctx cancel is sufficient.
Kafka producer	delivery.timeout.ms at connection config; per-message tuning not exposed.
RabbitMQ, NATS, MQTT	Publish is fire-and-ack; no server-side "abort my publish" mechanism. Broker either commits or doesn't.
SMTP	Short session; ctx cancel closes the connection.
HTTP notifications (Slack, MS Teams)	Short HTTP request; ctx cancel is sufficient.

Streaming / socket close is sufficient

Operations either stream (HTTP body) or hit a fast sync response — closing the socket immediately stops server-side work. No server-side abort mechanism is needed.

Category	Connectors
Object storage	S3, Azure Blob, Google Cloud Storage, OneLake, Azure IoT Hub
File	Local File, FTP, SMB
Cache	Redis
Cloud messaging	Kinesis, SNS, SQS, Google Pub/Sub

IoT / industrial-control

Sync TCP request-response protocols. Socket close on ctx expiry is instant server-side abort. Server holds no state after replying.

Modbus, OPC UA, OPC DA, S7, EtherNet/IP, BACnet, TwinCAT, CANopen, Melsec, Omron, MTConnect, Ignition, SiteWise, LoRaWAN, Sparkplug B.

Setting Connection.RequestTimeout

The typed RequestTimeout field on a connection is the single source of truth for Layer 5. It's a time.Duration — the standard Go duration type.

In connection config JSON:

{
  "requestTimeout": "30s"
}

Accepted formats: any string parseable by Go's time.ParseDuration — "30s", "5m", "1h30m", "500ms".

Precedence:

1. Connection.RequestTimeout (typed field, if set)
2. The connection's protocol-specific requestTimeoutSeconds config field (only used within Connect() — does not feed Layer 5)
3. Layer 5 drops out; the effective deadline comes from higher layers

When to set it:

• You have specific per-connection latency expectations (e.g., "no Snowflake query should exceed 5 min on this warehouse")
• You want a bound tighter than the pipeline's default 10-min ceiling without setting it on every node

When to leave it unset:

• The 10-min pipeline ceiling is fine for your workload
• You'd rather set the bound at the pipeline or node level so it's explicit per-run

Setting Connection.ConnectTimeout

Layer 6 bounds how long the connector spends opening the connection (TCP dial + TLS handshake + auth exchange). Applied once at Connect(), not per request. Independent from Layers 1–5.

Default: 10 seconds.

In connection config JSON:

{
  "connectTimeout": "15s"
}

When to raise it:

• Slow authentication mechanisms (Kerberos, some SSO flows)
• Distant cloud regions where TLS handshake dominates
• Backhaul networks with high latency

Layer composition examples

Case 1: Pipeline sets a tight cap, everything below is unset.

Pipeline: 60s
Node: unset
Function: unset
Connection: unset

Effective deadline: 60s (Layer 1 wins by construction).

Case 2: User sets a Snowflake connection cap of 5 min but the pipeline is 30 min.

Pipeline: 30 min
Node: unset
Function: unset
Connection: 5 min

Effective deadline: 5 min (Layer 5 tighter than Layer 1).

Case 3: A node explicitly overrides to 90s for a slow batch operation.

Pipeline: 30 min
Node: 90s
Function: unset
Connection: 5 min

Effective deadline: 90s (Layer 2 tightest).

Case 4: Nothing is set anywhere.

Pipeline: unset → 10 min default
Node: unset
Function: unset
Connection: unset

Effective deadline: 10 min (Layer 1 safety ceiling).

Common pitfalls

Setting timeoutMs on function config no longer overrides Layer 4. Function-level timeout was migrated to the typed Function.RequestTimeout field. Legacy config["timeoutMs"] is still honored as a fallback for migration convenience, but new configurations should use the typed field.

AWS Lambda's execution timeout is not observable via ctx. Lambda functions run to their configured maximum regardless of what MaestroHub's ctx says. If your Lambda is configured for 15 minutes and you set Connection.RequestTimeout: 30s, MaestroHub returns after 30s but you pay for the full Lambda duration.

Long-poll operations must fit within the composed budget. After the timeout-hierarchy refactor, SQS's Receive with waitTimeSeconds: 20 against a Connection.RequestTimeout of 15s cuts short at 15s. Set the pipeline/connection timeout to comfortably exceed the long-poll window plus a few seconds of headroom.