Google Pub/Sub Integration Guide

Connect to Google Cloud Pub/Sub to publish messages to topics and trigger pipelines from subscriptions. This guide covers connection setup, function configuration, and pipeline integration for Google Pub/Sub deployments.

Overview

The Google Pub/Sub connector enables integration with Google Cloud's fully managed messaging service, commonly used for event streaming, microservice communication, and real-time data pipelines. It provides:

• Message publishing to Pub/Sub topics with support for ordering keys, custom attributes, and dynamic payload templates
• Subscription-based pipeline triggers that start pipeline execution when messages arrive on a configured subscription
• At-least-once delivery guarantees with manual acknowledgment for reliable message processing
• Message ordering with ordering keys to ensure sequenced delivery within the same key
• Flow control with configurable maximum outstanding messages to prevent consumer overload
• Service account authentication using JSON key files for secure, automated access

Fully Managed

Google Cloud Pub/Sub is a serverless messaging service — no infrastructure provisioning is required. Connections are established via the Google Cloud API using a project ID and service account credentials.

Connection Configuration

Google Pub/Sub Connection Creation Fields

1. Profile Information

Field	Default	Description
Profile Name	-	A descriptive name for this connection profile (required, max 100 characters)
Description	-	Optional description for this Google Pub/Sub connection

2. Google Cloud Settings

Field	Default	Description
Project ID	-	Google Cloud project ID that contains your Pub/Sub topics and subscriptions (required)
Connection Timeout (seconds)	30	Timeout for establishing the connection (1–300 seconds)

Finding Your Project ID

Your project ID can be found in the Google Cloud Console at the top of the page or by running gcloud config get-value project in Cloud Shell. It is the unique identifier for your Google Cloud project (e.g., my-messaging-project-123).

3. Service Account Key

Field	Default	Description
Service Account Key JSON	-	JSON key file for a Google Cloud service account with Pub/Sub access. Leave empty when using the Pub/Sub Emulator or default application credentials.

You can provide the service account key by either uploading a JSON key file (drag-and-drop or file picker) or pasting the JSON content directly into the field.

To create a service account and generate a key:

Step 1: Create a Service Account

1. Open the Google Cloud Console IAM → Service Accounts
2. Select your project
3. Click + Create Service Account
4. Enter a name (e.g., maestrohub-pubsub) and an optional description
5. Click Create and Continue

Step 2: Grant Pub/Sub Permissions

Assign one of the following roles to the service account depending on your use case:

Role	Permissions	Recommended For
Pub/Sub Publisher	Publish messages to topics	Publish-only pipelines
Pub/Sub Subscriber	Consume messages from subscriptions	Subscribe-only pipelines
Pub/Sub Editor	Publish and subscribe	Pipelines that both publish and consume
Pub/Sub Admin	Full management of topics and subscriptions	Administrative use cases

Minimum Required Roles

For most use cases, assign Pub/Sub Editor. This grants both publish and subscribe permissions without allowing topic/subscription management. If your pipelines only publish, use Pub/Sub Publisher; if they only subscribe, use Pub/Sub Subscriber.

Step 3: Generate a JSON Key

1. In the service account list, click on your newly created service account
2. Navigate to the Keys tab
3. Click Add Key → Create new key
4. Select JSON as the key type
5. Click Create — the key file will download automatically

Key Security

The downloaded JSON key file contains credentials that grant access to your Pub/Sub resources. Store it securely and never commit it to version control. In MaestroHub, the key is encrypted and stored securely. On edit, it is displayed as masked — leave it unchanged to keep the stored value, or upload a new key to replace it.

4. Connection Labels

Field	Default	Description
Labels	-	Key-value pairs to categorize and organize this Google Pub/Sub connection (max 10 labels)

Example Labels

• env: prod – Environment
• team: data-engineering – Responsible team
• region: us-central1 – GCP region

Function Builder

Creating Google Pub/Sub Functions

Once you have a connection established, you can create reusable publish and subscribe functions:

1. Navigate to Functions → New Function
2. Select the desired function type (Publish or Subscribe)
3. Choose your Google Pub/Sub connection
4. Configure the function parameters

Select from two Google Pub/Sub function types: Publish for sending messages and Subscribe for receiving messages

Publish Function

Purpose: Publish messages to Google Pub/Sub topics with support for ordering keys, custom attributes, and dynamic payload templates. Use this for event broadcasting, data pipeline ingestion, and inter-service messaging.

Configuration Fields

Field	Type	Required	Default	Description
Topic	String	Yes	-	Pub/Sub topic name to publish to (e.g., my-topic). Do not include the full resource path. Supports ((parameter)) syntax.
Payload	String	Yes	-	Message payload content (JSON or text). Supports ((parameter)) syntax for dynamic content.
Ordering Key	String	No	-	Key for ensuring ordered delivery. Messages with the same ordering key are delivered in order. Supports ((parameter)) syntax.
Attributes	JSON	No	-	Custom message attributes as key-value string pairs for filtering or metadata (e.g., {"source": "iot-gateway", "version": "1.0"}). Supports ((parameter)) syntax.

Use Cases:

• Stream IoT sensor events to downstream analytics consumers
• Broadcast application events to multiple subscriber services
• Ingest time-series data into cloud data pipelines
• Forward operational alerts and notifications across services

Message Ordering

To use ordering keys, your Pub/Sub topic must have message ordering enabled at creation time. Enabling ordering on an existing topic is not supported. When ordering is enabled for a key, Pub/Sub guarantees sequential delivery for all messages sharing the same key.

---

Subscribe Function

Purpose: Subscribe to a Google Pub/Sub subscription to receive messages as they arrive. Subscribe functions are used as pipeline triggers — when a message arrives on the configured subscription, a new pipeline execution starts with the message payload and metadata.

Configuration Fields

Field	Type	Required	Default	Description
Subscription	String	Yes	-	Pub/Sub subscription name to consume from (e.g., my-subscription). Do not include the full resource path.
Max Outstanding Messages	Number	No	100	Maximum number of unprocessed messages held in memory at once (flow control). Set to 0 for no limit.
Auto Acknowledge	Boolean	No	false	When enabled, messages are acknowledged immediately upon delivery regardless of processing outcome (at-most-once). When disabled, messages are acknowledged only after successful pipeline execution (at-least-once, recommended).

Acknowledgment Behavior

• Auto Acknowledge disabled (recommended): MaestroHub acknowledges the message only after the pipeline completes successfully. If processing fails, the message is requeued and redelivered — ensuring at-least-once delivery. Use this for reliable event processing.
• Auto Acknowledge enabled: The message is acknowledged immediately upon delivery regardless of whether the pipeline succeeds. Faster, but messages may be lost if processing fails — at-most-once delivery. Use this only when occasional message loss is acceptable.

Use Cases:

• Trigger pipelines on incoming cloud events and notifications
• Process real-time data streams from IoT devices
• React to application events published by upstream services
• Consume task queues for asynchronous workload processing

---

Using Parameters

The ((parameterName)) syntax creates dynamic, reusable functions. Parameters are automatically detected from your configuration fields and can be configured with:

Configuration	Description	Example
Type	Data type validation	string, number, boolean, datetime, json, buffer
Required	Make parameters mandatory or optional	Required / Optional
Default Value	Fallback value if not provided	sensor-telemetry, us-central1, {}
Description	Help text for users	"Target topic name", "Device identifier"

Configure dynamic parameters for Publish functions with type validation, defaults, and descriptions

Parameter Availability

Template parameters are available for Publish functions (in topic, payload, ordering key, and attributes). Subscribe functions are event-driven triggers and do not accept runtime parameters — their subscription name is fixed at configuration time.

Pipeline Integration

Use the Google Pub/Sub functions you create here as nodes inside the Pipeline Designer. Drag them onto the canvas, bind parameters to upstream outputs or constants, and build event-driven cloud messaging flows without leaving the designer.

• Google Pub/Sub Trigger — Automatically start a pipeline when a message arrives on a Pub/Sub subscription
• Google Pub/Sub Publish — Publish a message to a Pub/Sub topic as a step within a running pipeline

For broader orchestration patterns that combine Google Pub/Sub with databases, REST, MQTT, or other connector steps, see the Connector Nodes page.

Common Use Cases

IoT Telemetry Ingestion

Scenario: Collect sensor readings from factory equipment and publish them to Google Pub/Sub for fan-out distribution to analytics, storage, and alerting consumers.

Publish Configuration:

• Topic: factory-telemetry
• Ordering Key: ((machineId)) — guarantees per-machine message order
• Payload:

{
  "machineId": "((machineId))",
  "temperature": ((temperature)),
  "vibration": ((vibration)),
  "timestamp": "((timestamp))"
}

• Attributes: {"plant": "chicago", "line": "assembly-1"}

Pipeline Integration: Connect after OPC UA or Modbus read nodes to continuously stream equipment telemetry into Google Pub/Sub.

---

Event-Driven Microservice Communication

Scenario: Trigger pipeline executions in response to application events published by upstream services (e.g., order created, payment processed, user registered).

Subscribe Configuration:

• Subscription: order-events-maestrohub
• Max Outstanding Messages: 50
• Auto Acknowledge: false

Pipeline Integration: Use the Subscribe trigger node to start a pipeline when an order event arrives, then route the event to an ERP system via REST, log it to PostgreSQL, and send a confirmation via SMTP.

---

Real-Time Stream Processing

Scenario: Consume raw events from a Pub/Sub subscription, enrich and transform the payload, and publish the result to a different topic for downstream analytics consumers.

Subscribe Trigger Configuration:

• Subscription: raw-events-subscription
• Trigger Mode: Always
• Auto Acknowledge: false

Publish Configuration (downstream node):

• Topic: enriched-events
• Ordering Key: ((sourceId))
• Payload: Transformed payload from upstream pipeline nodes

Pipeline Integration: Chain the Pub/Sub trigger with transform and enrichment nodes, then publish the result to a new topic — building a fully managed stream processing pipeline without dedicated infrastructure.

---

Cross-Service Notification Routing

Scenario: Consume alert events from a Pub/Sub subscription and route them to the correct notification channel based on severity.

Subscribe Configuration:

• Subscription: alerts-subscription
• Trigger Mode: Always
• Auto Acknowledge: false

Pipeline Integration: Connect to condition nodes that branch by severity — critical alerts route to MS Teams with full context, warnings are sent via SMTP, and all events are logged to MongoDB for audit purposes.

---

Deduplication with On Change Mode

Scenario: A Pub/Sub topic delivers sensor readings at a high frequency but you only want to act when the reading actually changes — avoiding redundant pipeline executions and downstream writes.

Subscribe Trigger Configuration:

• Subscription: temperature-readings-sub
• Trigger Mode: On Change
• Max Tracked Keys: 500 — track up to 500 distinct sensor IDs
• State TTL: 24h (Enterprise) — reset state for sensors silent for more than a day

Pipeline Integration: The trigger fires only when the incoming payload differs from the last processed payload for the same key, reducing unnecessary executions and downstream load.