MaestroHub and Databricks

Land plant data as files that Auto Loader turns into Delta tables, write rows through a SQL warehouse, or go through your streaming service, then read results back to the line.

Connectors

Databricks Storage · Databricks SQL

Start with

A · Files in a volume, loaded by Auto Loader

Network

Outbound HTTPS 443 for every route but F

Context Engine reaches Databricks through two connectors: Databricks Storage writes files to Unity Catalog volumes, and Databricks SQL runs queries and statements on a SQL warehouse. Most plants land Parquet files and let Auto Loader build the Delta tables, because nothing has to run between loads.

Find your route

Answer a few questions; the drawing below follows your answer.

Where does the data go?

Into DatabricksBack to the plant

How fresh must it be?

SecondsMinutes are fine

What can the plant reach?

The Databricks workspaceOnly our cloud storageOnly our streaming service

BWrite with SQL. MaestroHub has no direct streaming route into Databricks. An Execute call per batch lands rows in seconds while the SQL warehouse runs. For a steady high rate, put a streaming service in between (C).

Architecture

Context Engine runs next to your machines, and every route starts or ends there. Select a route to trace it.

Into Databricks

Back to the plant

• Set up in MaestroHub
• Your infrastructure
• Set up in Databricks

ALand files in a volume. Minutes; a continuous Auto Loader stream is faster. A job or Lakeflow pipeline runs Auto Loader. What each side sets up

Routes

A

Into Databricks

Land files in a volume

Start here for machine data

Freshness

Minutes; a continuous Auto Loader stream is faster

Databricks compute

A job or Lakeflow pipeline runs Auto Loader

Delivery

Auto Loader tracks loaded files and skips repeats

In MaestroHub

1. Batch rows with a Buffer or Aggregator node so each call carries many rows.
2. Add Convert to File with format Parquet and a declared column schema.
3. Write with Databricks Storage to a Unity Catalog volume path such as /Volumes/factory/raw/landing/((day))/((hour)).parquet, decoding the base64 output.

In Databricks

1. Create a Unity Catalog volume and grant the service principal WRITE VOLUME on it.
2. Create a job or Lakeflow pipeline whose Auto Loader reads the volume path into a Delta table.
3. Sign MaestroHub in with OAuth M2M for a service principal, or a personal access token.

Choose it when Plant data should land in Delta tables without a warehouse running for ingestion, and minutes are fine.

Look elsewhere when You need second-level freshness: use C.

Databricks Storage guideConvert to File

B

Into Databricks

Write with SQL

For records that change

Freshness

Per call, once the warehouse is running

Databricks compute

A SQL warehouse, billed while running

Delivery

Retried on failure; use MERGE so a retry cannot duplicate

In MaestroHub

1. Create a Databricks SQL connection with the workspace URL and the warehouse.
2. Build an Execute function with INSERT or MERGE and ((parameters)).
3. Batch rows with a Buffer or Aggregator node so each call carries many rows.

In Databricks

1. Set auto stop on the SQL warehouse.
2. Grant USE CATALOG, USE SCHEMA and MODIFY on the table to the service principal.

Choose it when Rows change after they land, or volumes are low.

Look elsewhere when Steady high-rate telemetry keeps the warehouse awake all day: use A or C.

Databricks SQL guide

C

Into Databricks

Through your streaming service

For second-level freshness

Freshness

Seconds

Databricks compute

A streaming job runs all the time

Delivery

Idempotent Kafka producer; Spark checkpoints track offsets

In MaestroHub

1. Create a Kafka, Azure Event Hubs or Kinesis connection.
2. Use Produce, Send Batch or Put Records per data set, keyed by asset.
3. Keep store-and-forward on the node.

In your infrastructure

1. A topic, event hub or stream per data set.

In Databricks

1. A streaming job that reads the topic and writes a Delta table, with its own checkpoint location.

Choose it when Freshness in seconds matters, or the stream already exists for other consumers.

Look elsewhere when Minutes are fine: A costs less because nothing runs between loads.

Kafka guideAzure Event Hubs guideKinesis guide

D

Into Databricks

Land files in your bucket

When the plant reaches only your cloud

Freshness

Minutes

Databricks compute

A job or Lakeflow pipeline runs Auto Loader

Delivery

Auto Loader tracks loaded files and skips repeats

In MaestroHub

1. Batch rows with a Buffer or Aggregator node so each call carries many rows.
2. Add Convert to File with format Parquet and a declared column schema.
3. Write with the S3, Azure Blob or GCS node, Input Format base64.

In your infrastructure

1. A bucket or container in your cloud account.

In Databricks

1. Register the bucket as an external location in Unity Catalog.
2. Point an Auto Loader job at the path.

Choose it when Your rules let the plant reach your cloud storage but not the workspace.

Look elsewhere when The plant can reach the workspace: A needs no bucket of your own.

Convert to FileS3 guideAzure Blob guideGCS guide

E

Back to the plant

Read results back

Close the loop on a schedule

Freshness

Your schedule

Databricks compute

A SQL warehouse runs each query

Delivery

Reads current state; safe to repeat

In MaestroHub

1. Build a Query function with a parameter for the last time read.
2. Pipeline: Schedule trigger, then the query, then publish to a UNS topic or write a setpoint through the machine's connector.

In Databricks

1. A small table or view of results with an updated_at column, with SELECT granted.

Choose it when Models or analytics in Databricks produce values the plant should use.

Look elsewhere when The plant must react when a job finishes: use F.

Databricks SQL guideSchedule trigger

F

Back to the plant

A job calls MaestroHub

React when a job finishes

Freshness

Seconds after the job run ends

Databricks compute

Whatever the job runs

Delivery

One HTTP call per notification; make the pipeline safe to run twice

In MaestroHub

1. Add a Webhook trigger with an auth header and an IP allowlist. A basic-auth destination sends an Authorization header the trigger can require.
2. Read the run result, then fetch the new values with E or tell the team.

In your infrastructure

1. Make the Context Engine webhook URL reachable over HTTPS from Databricks.

In Databricks

1. Create a notification destination of type Webhook with the Context Engine URL.
2. Add it to the job's notifications on success or failure.

Choose it when A job's outcome should start something on the line right away.

Look elsewhere when The plant accepts no inbound connections: poll with E.

Webhook trigger

No direct streaming route

MaestroHub does not stream rows into Databricks directly. For freshness in seconds, go through a streaming service (C), or write small batches with SQL (B) while the warehouse runs.

Compare the routes

Routes combine: most plants run one route in and one route back.

Route	Direction	Freshness	Databricks compute	Delivery	Network	Best for
	Into Databricks	Minutes; a continuous Auto Loader stream is faster	A job or Lakeflow pipeline runs Auto Loader	Auto Loader tracks loaded files and skips repeats	Outbound HTTPS 443 to the workspace	Telemetry and events into a bronze Delta layer
	Into Databricks	Per call, once the warehouse is running	A SQL warehouse, billed while running	Retried on failure; use MERGE so a retry cannot duplicate	Outbound HTTPS 443 to the workspace	Work orders, batch records, reference data
	Into Databricks	Seconds	A streaming job runs all the time	Idempotent Kafka producer; Spark checkpoints track offsets	Outbound to your streaming service	Fast dashboards, several consumers on one stream
	Into Databricks	Minutes	A job or Lakeflow pipeline runs Auto Loader	Auto Loader tracks loaded files and skips repeats	Outbound to your own bucket only	Lakes shared with other tools; strict network rules
	Back to the plant	Your schedule	A SQL warehouse runs each query	Reads current state; safe to repeat	Outbound HTTPS 443 to the workspace	Model scores, quality limits, targets
	Back to the plant	Seconds after the job run ends	Whatever the job runs	One HTTP call per notification; make the pipeline safe to run twice	Inbound HTTPS from Databricks to Context Engine	Acting when a retrain or scoring run finishes or fails

Related

• All data platforms
• Databricks Storage
• Databricks SQL
• Convert to File