Skip to content

Repository files navigation

Build status

Elastic connectors (aka crawlers)

search-icon

Use connectors to sync data from popular data sources to Elasticsearch.

This repository contains the source code for all Elastic connectors, also known as crawlers. Kibana 9.7 and later shows the UI page for managing connectors as Content crawlers.

Self-managed connectors run on your own infrastructure. For an overview of the deployment steps, refer to Self-managed connectors in the official Elastic documentation.

To get started quickly using Docker Compose, check out this README file.

Documentation

The main documentation lives in the Elastic docs. Check here for individual data source references and deployment information.

Since 8.12.0, you can manage connectors and sync jobs programmatically using APIs. Refer to the Connector API documentation.

Everything else concerning developing connectors is located in this repo.

Command-line interface

Learn about our CLI tool in docs/CLI.md.

Project structure

There are two main directories; the service code lives in the app/ directory and the SDK and framework in libs/.

The Connectors SDK is a dependency of the connectors application. You can use the Connectors SDK in and of itself to create your own data source implementations without needing to ingest data into Elasticsearch.

Service code app/

This service defines tasks such as running connectors and managing scheduling, syncs, and cleanup. This is shared code that is not used by individual connectors, but helps to coordinate and run a deployed instance/process.

SDK and framework libs/

The Connector SDK provides a framework that enables developers to build self-managed connectors. The framework implements common functionalities out of the box, so developers can focus on the logic specific to integrating their chosen data source.

The framework ensures compatibility, makes it easier for our team to review PRs, and helps out in the development process. When you build using our framework, we provide a pathway for the connector to be officially supported by Elastic.

Running a self-managed stack

This repo provides a set of scripts to allow a user to set up a full Elasticsearch, Kibana, and Connectors service stack using Docker. This is useful to get up and running with minimal effort, and provides a guided set of prompts for setup and configuration. For more information, instructions, and options, see the README file in the stack folder.

Framework use cases

The framework serves two distinct, but related use cases:

  • Customizing an existing Elastic connector client
  • Building a new self-managed connector

Guides for using the framework

Version compatibility with Elasticsearch

Note

Version compatibility is not checked when Elasticsearch is serverless (the connector service skips the check on startup).

The connector service compares its version to the configured Elasticsearch server on startup. If the versions are incompatible, the service exits and logs the mismatch. If the versions differ but are compatible, the service logs a warning and continues.

We strongly recommend running the connector service on the same stack version as Elasticsearch. Use this table to decide whether a deliberate version skew is supported.

Do not downgrade the connector service to an older release. If downgraded, connector configuration stored in Elasticsearch may no longer match what the older service expects and affected connectors will need to be recreated.

Situation Example connector version Example ES version Outcome
Versions match 8.15.1.0 8.15.1 OK
Connector has a build suffix 8.15.1.3 8.15.1 OK
ES patch is newer 8.15.0.0 8.15.1 Warning
ES minor is newer 8.14.2.0 8.15.0 Warning
ES major is newer 8.15.1.0 9.0.0 Fatal error
ES patch is older 8.15.1.0 8.15.0 Warning
ES minor is older 8.15.1.0 8.14.2 Fatal error
ES major is older 9.0.0.0 8.15.1 Fatal error

Releases

Used by

Contributors

Languages