Elasticsearch

Connect to Elasticsearch to search, index, retrieve, and delete documents in your pipelines. This guide covers connection setup, function configuration, and pipeline integration for both self-managed clusters and Elastic Cloud deployments.

Overview

The Elasticsearch connector enables integration with Elasticsearch clusters for full-text search, document storage, and real-time data retrieval. It provides:

• Full-text search with Elasticsearch Query DSL, including pagination support
• Document indexing for creating and updating documents with optional custom IDs
• Document retrieval by ID for fast single-document lookups
• Document deletion by ID for data lifecycle management
• Elastic Cloud support via Cloud ID for managed deployments
• Multiple authentication methods including Basic Auth and API Key
• TLS encryption with optional certificate verification for secure connections
• Template parameters for dynamic queries, index names, and document bodies

Connection Configuration

Creating an Elasticsearch Connection

Navigate to Connections → New Connection → Elasticsearch and configure the following:

Elasticsearch Connection Creation Fields

1. Profile Information

Field	Default	Description
Profile Name	-	A descriptive name for this connection profile (required, max 100 characters)
Description	-	Optional description for this Elasticsearch connection

2. Cluster Connection

Choose your deployment type:

Self-Managed Cluster

Field	Default	Description
Addresses	http://localhost:9200	Comma-separated list of Elasticsearch node URLs (e.g., http://node1:9200,http://node2:9200)

Elastic Cloud

Field	Default	Description
Cloud ID	-	The Cloud ID from your Elastic Cloud deployment. Found in Elastic Cloud Console → Deployment → Manage

Deployment Mode

• Self-Managed: Use for on-premise or self-hosted Elasticsearch clusters. Provide one or more node addresses.
• Elastic Cloud: Use for Elastic Cloud managed deployments. The Cloud ID contains all connection details encoded.

3. Authentication

Choose your authentication method:

No Authentication

No additional fields required. Use only for development clusters without security enabled.

Basic Authentication

Field	Default	Description
Username	-	Elasticsearch username with appropriate permissions
Password	-	Password for the Elasticsearch user. Masked on edit; leave empty to keep stored value

API Key Authentication

Field	Default	Description
API Key	-	Base64-encoded API key for authentication. Generate in Kibana → Stack Management → API Keys

Security Notice

API Key and Basic Authentication are mutually exclusive. Use API Keys for production deployments as they provide finer-grained access control and can be rotated without changing user passwords.

4. TLS/Security Settings

Field	Default	Description
Enable TLS	false	Enable TLS/SSL encryption for the connection

(Only displayed when Enable TLS is checked)

Field	Default	Description
Skip Certificate Verification	false	Skip TLS certificate verification (not recommended for production)
CA Certificate	-	PEM-encoded CA certificate for verifying the Elasticsearch server certificate

Security Notice

Enabling Skip Certificate Verification disables TLS certificate validation. Use only in trusted development environments, never in production.

5. Connection Labels

Field	Default	Description
Labels	-	Key-value pairs to categorize and organize this Elasticsearch connection (max 10 labels)

Example Labels

• env: prod – Environment
• cluster: logs – Cluster purpose
• region: us-east-1 – Deployment region

Notes

• Required Fields: Profile Name and either Addresses or Cloud ID must be provided.
• Address Format: Each address must include the protocol (http:// or https://).
• Multi-Node: For self-managed clusters, provide multiple node addresses for automatic failover.
• TLS Port: When TLS is enabled, Elasticsearch typically uses port 9243 for Elastic Cloud or 9200 with HTTPS for self-managed.

Function Builder

Creating Elasticsearch Functions

Once you have a connection established, you can create reusable Elasticsearch operation functions:

1. Navigate to Functions → New Function
2. Select the desired function type (Search, Index, Get, or Delete)
3. Choose your Elasticsearch connection
4. Configure the function parameters

Search Function

Purpose: Execute full-text searches using Elasticsearch Query DSL. Use this for complex queries, aggregations, filtering, and paginated results retrieval.

Configuration Fields

Field	Type	Required	Default	Description
Index	String	Yes	-	The index name or pattern to search (e.g., logs-*, products). Supports wildcards for multi-index searches.
Query	JSON	Yes	-	Elasticsearch Query DSL in JSON format. Can be a full query object or just the query clause.
Size	Number	No	10	Maximum number of documents to return (pagination).
From	Number	No	0	Offset from the first result for pagination.

Query DSL Examples

Match all documents:

{
  "match_all": {}
}

Full-text search:

{
  "match": {
    "message": "error"
  }
}

Boolean query with filters:

{
  "bool": {
    "must": [
      { "match": { "status": "active" } }
    ],
    "filter": [
      { "range": { "timestamp": { "gte": "now-1h" } } }
    ]
  }
}

Use Cases:

• Search log entries for errors or specific patterns
• Query product catalogs with filters and sorting
• Retrieve time-series data with range queries
• Perform aggregations for analytics dashboards

---

Index Function

Purpose: Create or update documents in an Elasticsearch index. Use this for storing new data, updating existing records, or bulk data ingestion.

Configuration Fields

Field	Type	Required	Default	Description
Index	String	Yes	-	The target index name for the document (e.g., logs-2024, products)
Body	JSON	Yes	-	The document content in JSON format. Supports template parameters.
Document ID	String	No	-	Optional document ID. If omitted, Elasticsearch auto-generates a unique ID. Provide for upsert operations.

Document Body Examples

Log entry:

{
  "timestamp": "((timestamp))",
  "level": "((level))",
  "message": "((message))",
  "service": "pipeline-engine"
}

Sensor reading:

{
  "sensor_id": "((sensorId))",
  "temperature": ((temperature)),
  "humidity": ((humidity)),
  "recorded_at": "((timestamp))"
}

Use Cases:

• Store processed pipeline data for analysis
• Index IoT sensor readings with timestamps
• Create audit logs from pipeline executions
• Sync data from other systems into Elasticsearch

---

Get Function

Purpose: Retrieve a single document by its ID. Use this for fast, direct lookups when you know the exact document identifier.

Configuration Fields

Field	Type	Required	Default	Description
Index	String	Yes	-	The index containing the document
Document ID	String	Yes	-	The unique ID of the document to retrieve

Response Fields

Field	Description
_id	The document ID
_index	The index name
found	Boolean indicating if document exists
_source	The document content (if found)
_version	Document version number

Use Cases:

• Retrieve configuration documents by known ID
• Look up user or device records
• Fetch cached data stored with deterministic IDs
• Validate document existence before updates

---

Delete Function

Purpose: Remove a document from an Elasticsearch index by its ID. Use this for data cleanup, lifecycle management, or removing invalid records.

Configuration Fields

Field	Type	Required	Default	Description
Index	String	Yes	-	The index containing the document to delete
Document ID	String	Yes	-	The unique ID of the document to delete

Response Fields

Field	Description
_id	The document ID
_index	The index name
result	deleted if successful, not_found if document didn't exist

Use Cases:

• Remove expired or obsolete records
• Delete processed items from queue indices
• Clean up test data
• Implement data retention policies

---

Using Parameters

The ((parameterName)) syntax creates dynamic, reusable functions. Parameters are automatically detected from your configuration fields and can be configured with:

Configuration	Description	Example
Type	Data type validation	string, number, boolean, datetime, json, buffer
Required	Make parameters mandatory or optional	Required / Optional
Default Value	Fallback value if not provided	logs-*, 10, {}
Description	Help text for users	"Index name pattern", "Search query"