MaestroHub and BigQuery
Send plant data to BigQuery through Pub/Sub or as files in Cloud Storage, start Dataflow jobs from a pipeline, and read results back to the line.
- Connectors
- Google Pub/Sub · GCS · Cloud Dataflow · BigQuery
- Start with
- A · Pub/Sub with a BigQuery subscription
- Network
- Outbound HTTPS 443 to Google Cloud
Rows reach BigQuery through Google Cloud services that Context Engine already connects to: a Google Pub/Sub topic with a BigQuery subscription writes each message as a row, and GCS holds files that BigQuery loads. The BigQuery connector reads results back.
Find your route
Answer a few questions; the drawing below follows your answer.
AThrough Pub/Sub. Continuous data that should be queryable within seconds, with no job to run.
Architecture
Context Engine runs next to your machines, and every route starts or ends there. Select a route to trace it.
- Set up in MaestroHub
- Your infrastructure
- Set up in BigQuery
Routes
- Freshness
- Seconds
- BigQuery compute
- No job runs; billed by volume
- Delivery
- At least once; keep a message ID column to remove duplicates
In MaestroHub
- Create a Google Pub/Sub connection with a service account key.
- Add a Publish node whose JSON fields match the table columns.
- Keep store-and-forward on the node.
In BigQuery
- Create the table.
- Create a BigQuery subscription on the topic that uses the table schema.
- Grant the Pub/Sub service agent BigQuery Data Editor on the table.
Choose it when Continuous data that should be queryable within seconds, with no job to run.
Look elsewhere when Large backfills from a historian: use B.
- Freshness
- Your load schedule
- BigQuery compute
- Batch loads use free shared slots; external tables bill per query
- Delivery
- Partition by path so a rerun replaces one partition
In MaestroHub
- Batch rows with a Buffer or Aggregator node so each call carries many rows.
- Add Convert to File with format Parquet and a declared column schema.
- Write with the GCS node, Input Format
base64, to a dated path.
In BigQuery
- Grant the service account Storage Object Creator on the bucket.
- Schedule a Cloud Storage transfer into the table, or define an external BigLake table on the path.
Choose it when Large volumes where minutes or hours are fine.
Look elsewhere when You need seconds: use A.
- Freshness
- Your batch schedule
- BigQuery compute
- Dataflow workers while the job runs
- Delivery
- Follow job state and metrics from the pipeline
In MaestroHub
- Create a Google Cloud Dataflow connection whose identity has
roles/dataflow.developer. - Pipeline: a trigger, then Launch a Flex or classic template with parameters, then follow its state and alert on failure.
In BigQuery
- A Google-provided or custom template that writes BigQuery.
Choose it when A load or transform should start when the plant finishes a batch or a shift.
Look elsewhere when You only need to land rows: A or B is simpler.
- Freshness
- Your schedule
- BigQuery compute
- Each query bills the bytes it scans
- Delivery
- Reads current state; safe to repeat
In MaestroHub
- Build a Query function. A result returns up to 50,000 rows or 40 MB by default.
- Pipeline: Schedule trigger, then the query, then publish to a UNS topic or write a setpoint through the machine's connector.
In BigQuery
- A small, partitioned results table so each query scans little, with BigQuery Data Viewer granted.
Choose it when Analytics or models in BigQuery produce values the plant should use.
Look elsewhere when Large exports: write them to Cloud Storage and read the files instead.
The BigQuery connector runs queries; this page does not cover writing rows through it. Rows go in through Pub/Sub (A) or files (B).
Compare the routes
Routes combine: most plants run one route in and one route back.
| Route | Direction | Freshness | BigQuery compute | Delivery | Network | Best for |
|---|---|---|---|---|---|---|
| Into BigQuery | Seconds | No job runs; billed by volume | At least once; keep a message ID column to remove duplicates | Outbound HTTPS 443 to Google Cloud | Telemetry and events | |
| Into BigQuery | Your load schedule | Batch loads use free shared slots; external tables bill per query | Partition by path so a rerun replaces one partition | Outbound HTTPS 443 to Google Cloud | Historian backfills, large daily volumes | |
| Into BigQuery | Your batch schedule | Dataflow workers while the job runs | Follow job state and metrics from the pipeline | Outbound HTTPS 443 to Google Cloud | Transforms and backfills started and watched from MaestroHub | |
| Back to the plant | Your schedule | Each query bills the bytes it scans | Reads current state; safe to repeat | Outbound HTTPS 443 to Google Cloud | Predictions, targets, master data |