AWS S3 Integration Guide
Use MaestroHub's S3 connector to work with AWS S3 or S3‑compatible object storage (MinIO, Wasabi, DigitalOcean Spaces). This guide covers connection setup, function authoring, and pipeline integration.
Overview
The S3 connector provides:
- AWS S3 and S3‑compatible support with region/endpoint controls and path‑style addressing
- Reusable functions to fetch and write objects with templates and overrides
- Secure credential handling with masked edits and server‑side encryption (SSE‑S3 and SSE‑KMS)
- Operational limits for discovery and maximum file size to protect pipelines
Connection Configuration
Creating an AWS S3 Connection
From Connections → New Connection → AWS S3, configure the fields below.
S3 Connection Creation Fields
1. Profile Information
| Field | Default | Description |
|---|---|---|
| Profile Name | - | A descriptive name for this connection profile (required, max 100 characters) |
| Description | - | Optional description for this S3 connection |
2. S3 Configuration
| Field | Default | Description |
|---|---|---|
| Region | - | AWS region (e.g., us-east-1) or Custom region string for S3‑compatible services – required |
| Bucket | - | Target bucket name (3–63 chars, lowercase, DNS‑compliant; cannot be an IP address) – required |
| Prefix | - | Optional base path prefix used as a security boundary for all operations |
3. AWS Credentials
| Field | Default | Description |
|---|---|---|
| Access Key ID | - | Required. Masked on edit; leave empty to keep stored value |
| Secret Access Key | - | Required. Masked on edit; leave empty to keep stored value |
| Session Token | - | Optional STS token. Masked on edit; leave empty to keep stored value |
4. Advanced
| Field | Default | Description |
|---|---|---|
| Discovery Limit | 2000 | Maximum objects to list during discovery (1–100000) – required |
| Max File Size (MB) | 25 | Maximum allowed object size for functions (1–25) – required |
| Connection Timeout (seconds) | 30 | Timeout for S3 API calls (5–300) – required |
5. Server‑Side Encryption
| Field | Default | Description |
|---|---|---|
| Enable Server‑Side Encryption | true | Enable SSE for all writes by default |
| Encryption Type | AES256 | AES256 (SSE‑S3), aws:kms (SSE‑KMS), or none – required when SSE is enabled |
| KMS Key ID | - | Required when Encryption Type = aws:kms. Masked on edit; leave empty to keep stored value |
6. S3‑Compatible Services
| Field | Default | Description |
|---|---|---|
| Quick Preset | Custom | MinIO, Wasabi, DigitalOcean Spaces, or Custom presets to apply recommended flags |
| Custom Endpoint | - | Optional endpoint URL for S3‑compatible services (e.g., http://localhost:9000). Leave empty for AWS S3 |
| Force Path Style | false | Use path‑style addressing (often required for MinIO) |
| Disable SSL | false | Disable HTTPS (local/dev only) |
7. Connection Labels
| Field | Default | Description |
|---|---|---|
| Labels | - | Key‑value pairs to categorize and organize this S3 connection (max 10 labels) |
Example Labels
environment: productionteam: data-platformstorage: s3region: us-east-1
- Bucket validation: Must start/end with letter or number; allowed chars are letters, numbers, hyphens, and periods. Disallow sequences
..,.-,-.. - Security: Credentials and KMS Key IDs are stored encrypted and displayed as masked on edit. Leave fields empty to keep stored values.
- S3‑compatible tips: Use Quick Preset; enable Path Style for MinIO; consider Disable SSL for local development only.
- Limits: Discovery Limit and Max File Size protect pipelines from large scans/uploads.
Function Builder
Creating S3 Functions
After saving the connection:
- Go to Functions → New Function
- Choose S3 Fetch or S3 Write as the function type
- Select the S3 connection profile
- Configure object keys, limits, and overrides
Design reusable S3 fetch and write operations with object key templates and overrides
Fetch Object
Purpose: Retrieve objects or lists of matching objects from the configured bucket/prefix using exact keys, wildcards, regex, or templates.
Configuration Fields
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| Object Key | String | Yes | - | Object path. Supports exact (data/report.csv), wildcards (logs/2024-*.log), regex (data/\\d{4}-\\d{2}-\\d{2}.csv), and templates (((date))/report.xlsx) |
| Timeout (ms) | Number | No | 60000 | Operation timeout (1000–600000) |
| Max File Size (MB) | Number | No | - | Overrides connection max (1–25) |
| Discovery Limit | Number | No | - | Overrides connection limit (1–100000) |
| Version ID | String | No | - | Specific object version for versioned buckets |
Use Cases: Retrieve daily reports, collect log partitions, hydrate datasets into pipelines
Write Object
Purpose: Upload objects to the configured bucket/prefix with optional overwrite policy, metadata/tags, storage class, and encryption overrides.
Configuration Fields
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| Object Key | String | Yes | - | Target path (max 1024). Supports templates e.g., data/output_((date)).csv |
| Data | String | Yes | - | Content to upload. Supports plain text and base64‑encoded content |
| Content Type | String | No | - | MIME type (auto‑detected if omitted) |
| Overwrite Existing | Boolean | No | true | If false and object exists, a timestamped filename is created |
| Storage Class | Enum | No | STANDARD | One of: STANDARD, REDUCED_REDUNDANCY, STANDARD_IA, ONEZONE_IA, INTELLIGENT_TIERING, GLACIER, DEEP_ARCHIVE, GLACIER_IR |
| Server‑Side Encryption | String | No | - | Optional override: AES256, aws:kms, or none (inherits connection if omitted). For aws:kms, provide KMS Key ID |
| Cache Control | String | No | - | Value for HTTP Cache-Control header |
| Metadata | Object | No | {} | User metadata as key/value pairs |
| Tags | Object | No | {} | Tag set as key/value pairs (up to 10 tags) |
| Timeout (ms) | Number | No | 60000 | Operation timeout (1000–600000) |
| Max File Size (MB) | Number | No | - | Overrides connection max (1–25) |
| Generate Presigned URL | Boolean | No | false | Return a presigned URL for the uploaded object |
| Presigned URL Expiry (s) | Number | No | 3600 | Expiry time for presigned URL (60–604800) |
Use Cases: Export pipeline results, publish artifacts, archive backups with lifecycle tiers
Using Parameters
Use ((parameterName)) in object keys, content templates, or metadata values to expose parameters for validation and runtime binding.
| Configuration | Description | Example |
|---|---|---|
| Type | Validate incoming values | string, number, boolean, datetime, json, buffer |
| Required | Enforce presence | Required / Optional |
| Default Value | Provide fallbacks | 'reports', '{}', NOW() |
| Description | Document intent | "Object key suffix (YYYY‑MM‑DD)", "Custom metadata JSON" |
Parameter validation, defaults, and helper text for S3 object keys and payloads
Pipeline Integration
Use the S3 connection functions you configure here as nodes inside the Pipeline Designer to move files between systems. Drag in the fetch or write node, bind parameters to upstream outputs or constants, and tune retries or error branches.
For broader orchestration patterns that mix S3 with SQL, REST, or MQTT steps, see the Connector Nodes page.
S3 fetch node with connection, function, and parameter bindings
Common Use Cases
Data Lake Ingestion
Ingest CSV/JSON objects from S3 into analytical stores or trigger downstream normalization pipelines.
Backup and Restore
Store pipeline outputs and model artifacts with versioning; retrieve specific versions for rollback.
Report Distribution
Publish generated reports to S3 and optionally return presigned URLs for external access.