MaestroHub and Snowflake

Stream plant data into Snowflake tables, land it as files or through Kafka, and bring results back to the line on a schedule or when an alert fires.

Connectors

Snowflake Streaming · Snowflake (SQL)

Start with

A · Stream rows, no warehouse running

Network

Outbound HTTPS 443 for every route but F

Snowflake has two connectors in Context Engine. Snowflake Streaming loads rows continuously through Snowpipe Streaming, with no warehouse running; Snowflake (SQL) runs queries and statements through a warehouse. Both are built into Context Engine, so nothing else is installed in the plant. The other routes pass through storage or Kafka you already run, or let Snowflake call Context Engine.

Find your route

Answer a few questions; the drawing below follows your answer.

Where does the data go?

Into SnowflakeBack to the plant

How fresh must it be?

SecondsMinutes are fine

What can the plant reach?

Snowflake directlyOnly our cloud storageOnly our Kafka

AStream rows. Continuous data that should be queryable within seconds, without paying for a running warehouse.

Architecture

Context Engine runs next to your machines, and every route starts or ends there. Select a route to trace it.

Into Snowflake

Back to the plant

• Set up in MaestroHub
• Your infrastructure
• Set up in Snowflake

AStream rows. Queryable within seconds. Serverless ingest, no warehouse. What each side sets up

Routes

A

Into Snowflake

Stream rows

Start here for machine data

Freshness

Queryable within seconds

Snowflake compute

Serverless ingest, no warehouse

Delivery

Exactly once

In MaestroHub

1. Create a Snowflake Streaming connection: account, service user, key pair. The connector is built into Context Engine; nothing else is installed.
2. Add a Stream Rows node after your trigger. Batch rows with a Buffer or Aggregator node so each call carries many rows.
3. Keep store-and-forward on the node. Rows written during an outage are replayed later and never written twice.
4. Schedule a Channel Status check to catch rows Snowflake rejected.

In Snowflake

1. Create a service user with an RSA public key.
2. Grant the user's default role USAGE on the database and schema, and INSERT and SELECT on the tables.
3. Create the target table, or let the connector create it.

Choose it when Continuous data that should be queryable within seconds, without paying for a running warehouse.

Look elsewhere when You need updates or MERGE: use B. Your rules forbid the plant reaching a SaaS endpoint: use C or D.

Snowflake Streaming guideStore-and-forward

B

Into Snowflake

Write with SQL

For records that change

Freshness

Per call, once the warehouse is awake

Snowflake compute

A virtual warehouse resumes for each batch

Delivery

Retried on failure; use MERGE so a retry cannot duplicate

In MaestroHub

1. Create a Snowflake connection with a warehouse, using a password or a key pair.
2. Build a Write function, which detects the schema and can create the table, or an Execute function with your MERGE.
3. Batch rows with a Buffer or Aggregator node so each call carries many rows.

In Snowflake

1. Grant the role USAGE on the warehouse, database and schema.
2. Grant INSERT and UPDATE, and CREATE TABLE if the function creates the table.
3. Set a short AUTO_SUSPEND on the warehouse to cap cost.

Choose it when Rows are updated in place, volumes are low, or you need SQL the streaming API does not run.

Look elsewhere when High-frequency telemetry: a warehouse kept awake all day costs credits all day. Use A.

Snowflake (SQL) guide

C

Into Snowflake

Land files

For bulk volume and a shared data lake

Freshness

About a minute after a file lands

Snowflake compute

Snowpipe, serverless

Delivery

Snowpipe remembers loaded files and skips repeats

In MaestroHub

1. Batch rows with a Buffer or Aggregator node so each call carries many rows.
2. Add Convert to File with format Parquet and a declared column schema.
3. Write with the S3, Azure Blob or GCS node, Input Format base64, to a path such as raw/telemetry/((day))/((hour)).parquet.

In your infrastructure

1. A bucket or container with event notifications switched on.

In Snowflake

1. Create a storage integration and an external stage on the bucket.
2. Create a pipe with AUTO_INGEST = TRUE that runs COPY INTO the table.
3. Point the bucket's event notifications at the pipe.

Choose it when Minutes are fine and volume is large, or the same files should also feed other tools.

Look elsewhere when You need second-level freshness: use A.

Convert to FileS3 guideAzure Blob guideGCS guide

D

Into Snowflake

Through your Kafka

When Kafka is already your backbone

Freshness

Seconds

Snowflake compute

Snowpipe Streaming, no warehouse

Delivery

Idempotent producer; the Kafka connector tracks offsets

In MaestroHub

1. Create a Kafka connection to your brokers.
2. Add a Produce node per data set, keyed by asset so each asset stays in order.
3. Keep store-and-forward on the node.

In your infrastructure

1. Kafka Connect running the Snowflake Connector for Kafka in Snowpipe Streaming mode, with each topic mapped to a table.

In Snowflake

1. A service user with a key pair for the Kafka connector, with grants on the target schema.

Choose it when Your platform team owns ingestion through Kafka, or other consumers read the same stream.

Look elsewhere when You do not run Kafka already. A gives the same result with one system fewer to operate.

Kafka guide

E

Back to the plant

Read results back

Close the loop on a schedule

Freshness

Your schedule, for example every minute

Snowflake compute

A warehouse runs each query

Delivery

Reads current state; safe to repeat

In MaestroHub

1. Build a Query function with a parameter for the last time read.
2. Pipeline: Schedule trigger, then the query, then publish to a UNS topic or write a setpoint through the machine's connector.

In Snowflake

1. Expose a small view of results with an updated_at column.
2. Grant SELECT on the view and USAGE on a small warehouse.

Choose it when Analytics or models in Snowflake produce values the plant should use.

Look elsewhere when The plant must react within seconds of an event in Snowflake: use F.

Snowflake (SQL) guideSchedule trigger

F

Back to the plant

Snowflake calls MaestroHub

React to events in Snowflake

Freshness

Seconds after the alert fires

Snowflake compute

The alert condition runs on a warehouse or serverless

Delivery

One HTTP call per notification; make the pipeline safe to run twice

In MaestroHub

1. Add a Webhook trigger with an auth header and an IP allowlist.
2. Read the payload and act: publish to the UNS, post to Teams or Slack, or write a setpoint.

In your infrastructure

1. Make the Context Engine webhook URL reachable over HTTPS from Snowflake.

In Snowflake

1. Create a notification integration with TYPE = WEBHOOK, the webhook URL, and a header that carries the trigger's secret.
2. Create an alert or task that calls SYSTEM$SEND_SNOWFLAKE_NOTIFICATION.

Choose it when A condition detected in Snowflake should start a shop-floor action right away.

Look elsewhere when The plant accepts no inbound connections: use E and poll on a short schedule.

Webhook trigger

Compare the routes

Routes combine: most plants run one route in and one route back.

Route	Direction	Freshness	Snowflake compute	Delivery	Network	Best for
	Into Snowflake	Queryable within seconds	Serverless ingest, no warehouse	Exactly once	Outbound HTTPS 443 to Snowflake	Telemetry, counts, quality results, events
	Into Snowflake	Per call, once the warehouse is awake	A virtual warehouse resumes for each batch	Retried on failure; use MERGE so a retry cannot duplicate	Outbound HTTPS 443 to Snowflake	Work orders, shift reports, genealogy, reference data
	Into Snowflake	About a minute after a file lands	Snowpipe, serverless	Snowpipe remembers loaded files and skips repeats	Outbound to your own bucket only	High volume, historian backfills, one lake for several tools
	Into Snowflake	Seconds	Snowpipe Streaming, no warehouse	Idempotent producer; the Kafka connector tracks offsets	Outbound to your Kafka brokers	A central streaming platform with several consumers
	Back to the plant	Your schedule, for example every minute	A warehouse runs each query	Reads current state; safe to repeat	Outbound HTTPS 443 to Snowflake	Predictions, quality limits, targets, recipes
	Back to the plant	Seconds after the alert fires	The alert condition runs on a warehouse or serverless	One HTTP call per notification; make the pipeline safe to run twice	Inbound HTTPS from Snowflake to Context Engine	Conditions found in analytics that should start an action on the line

Related

• All data platforms
• Snowflake Streaming
• Snowflake (SQL)
• Store-and-forward