OpenCrawling

OpenCrawling is the reference Java and Spring Framework implementation of the Open Ingestion Standard (OIS). It provides a secure, decoupled, and vendor-neutral enterprise data integration platform leveraging modern Java 25 features (such as Structured Concurrency and Virtual Threads), Spring Boot, Spring AI, and vector search infrastructure to orchestrate data flows from various repository connectors to vector search outputs.

---

🎬 Admin UI Screencast Walkthrough

---

Architecture Diagram

The diagram below shows the high-level architecture of OpenCrawling, highlighting the newly decoupled, stateless embedding microservice and transformation connectors:

graph TD
    subgraph UI
        UI_App[Admin React UI - oc-admin-ui]
    end

    subgraph PlatformRuntime [OpenCrawling Ingestion Runtime - oc-runtime]
        Runtime_Node([OpenCrawling Ingestion Runtime])
        Core[Core Ingestion Engine - oc-core]
        FS_Conn[Filesystem Repository - oc-filesystem-repository-connector]
        ACS_Conn[Alfresco Repository - oc-alfresco-repository-connector]
        APS_Conn[Alfresco Process Services - oc-aps-repository-connector]
        CMIS_Conn[OASIS CMIS Repository - oc-cmis-repository-connector]
        Doxis_Repo_Conn[Doxis 4 Repository - oc-doxis-repository-connector]
        SharePoint_Conn[SharePoint Online Repository - oc-sharepoint-repository-connector]
        MFiles_Repo_Conn[M-Files Vault - oc-mfiles-repository-connector]
        FileNet_Repo_Conn[IBM FileNet P8 Object Store - oc-filenet-repository-connector]
        Documentum_Repo_Conn[OpenText Documentum Repository - oc-documentum-repository-connector]
        OpenArch_Repo_Conn[Hyland OpenArch Repository - oc-openarch-repository-connector]
        Iceberg_Conn[Iceberg Repository - oc-iceberg-repository-connector]
        Flowable_Conn[Flowable Repository - oc-flowable-repository-connector]
        Camunda_Conn[Camunda Repository - oc-camunda-repository-connector]
        StormCrawler_Conn[StormCrawler Web Engine - oc-stormcrawler-repository-connector]
        gRPC_Push_Conn[gRPC Push Ingestion Gateway - oc-grpc-push-connector]
        
        Ing_Cons[Ingestion Consumer - IngestionConsumer]
        Tika[Apache Tika Text Extractor]
        Chunker[Token Chunker]
        
        Writer_Cons[Vector Store Writer - VectorStoreWriterConsumer]
        Precompute_Model[PrecomputedEmbeddingModel]
        Vec_Conn[Vector Stores & Fan-Out: pgvector / Solr 10 / Vespa / Qdrant / Milvus / OpenSearch / Luxir / OpenArch / Apache SeaTunnel]

        McpServer[Secure MCP Server - McpVectorServer]
        
        Core --> FS_Conn
        Core -->|Publish IngestionMessage| Ingest_Topic[(Kafka Topic: opencrawling-ingestion)]
        gRPC_Push_Conn -->|Push IngestionMessage| Ingest_Topic
        
        Ingest_Topic -->|Consume IngestionMessage| Ing_Cons
        Ing_Cons -->|Extract Text| Tika
        Tika --> Chunker
        Chunker -->|Publish Chunks| Chunk_Topic[(Kafka Topic: opencrawling-chunks)]
        
        Embed_Topic[(Kafka Topic: opencrawling-embedded)] -->|Consume EmbeddedMessage| Writer_Cons
        Writer_Cons --> Precompute_Model
        Precompute_Model --> Vec_Conn

        McpServer -->|Queries - Enforces ACLs| Vec_Conn
    end

    subgraph Embedding Service [OpenCrawling Embedding Service - oc-embedding-service]
        Embed_Cons[Embedding Consumer - EmbeddingConsumer]
        Model_Factory[EmbeddingModelFactory]
        
        Chunk_Topic -->|Consume ChunkMessage| Embed_Cons
        Embed_Cons --> Model_Factory
        Embed_Cons -->|Publish Embedded| Embed_Topic
    end

    subgraph Infrastructure [Docker Containers]
        PG[(PostgreSQL + pgvector)]
        Redis[(Redis Cache & Session)]
        Ollama[Ollama AI Embeddings]
        Kafka_Broker[Apache Kafka Broker]
    end

    subgraph External [AI Clients]
        LLM[AI Client / LLM Agent]
        OpenAI[OpenAI Platform]
    end

    UI_App -->|REST API| Runtime_Node
    Vec_Conn -->|Vectors| PG
    Runtime_Node -->|Job Cache| Redis
    
    Model_Factory -->|Local Inference| Ollama
    Model_Factory -->|Cloud Inference| OpenAI
    
    Ingest_Topic --> Kafka_Broker
    Chunk_Topic --> Kafka_Broker
    Embed_Topic --> Kafka_Broker

    LLM -->|Model Context Protocol| McpServer

Loading

---

Administration Dashboard (oc-admin-ui)

The oc-admin-ui provides a modern web-based administration console to monitor and configure your ingestion jobs.

UI Screenshots

📊 Telemetry Dashboard

Real-time graphs monitoring job success rates, Kafka queue load, active crawling threads, and index ingestion speed.

📋 Job Pipeline Scheduler

Schedule, monitor, start, and pause ingestion crawl tasks. Review document indexing status reports.

📁 Connector Configurations

Manage endpoints and credentials for repositories (SharePoint, S3, Filesystem), output vectors, and transformation engines.

⚙️ Ingestion & Embedding Mappings

Configure target models (e.g., Ollama, OpenAI) and tune text chunk sizes/overlap boundaries dynamically.

🪵 Real-Time Ingestion Logs

Inspect live Java logging streams and Kafka consumer offsets to troubleshoot connector execution.

---

💻 OpenCrawling CLI (oc)

The OpenCrawling CLI (oc-cli, binary oc) is a terminal-native, enterprise-grade command-line interface built with Java 25, Picocli, and JLine for DevSecOps automation, CI/CD pipeline integration, connector health inspection, offline OIS schema validation, and terminal ingestion management.

🚀 Quick Start

# Build the CLI executable JAR
mvn clean package -pl oc-cli

# Run CLI commands
java --enable-preview -jar oc-cli/target/oc-cli-1.0.4-SNAPSHOT.jar system status

🐳 Docker Container Execution

docker run --rm -it ghcr.io/opencrawling/oc-cli:latest --help
docker run --rm --network host ghcr.io/opencrawling/oc-cli:latest system status

🍺 Homebrew Installation (macOS / Linux)

brew tap opencrawling/tap
brew trust opencrawling/tap   # Required on Homebrew 7+
brew install opencrawling/tap/oc
oc --version

📋 Command Summary Matrix

Command	Subcommands	Description
oc job	list, start, status, pause, stop, delete, create	Manage crawling and data ingestion jobs.
oc connector	list, check, create, delete	Inspect connector health and registry definitions.
oc system	status, logs, settings, throughput	Query runtime component health and settings.
oc aiops	rca, spans, errors, metrics	Query OpenTelemetry traces and AI Root Cause Analysis.
oc copilot	narrativize, mock	Generate natural language Mustache templates from schemas.
oc archetype	init	Scaffold custom repository/output connector Maven projects.
oc schema	validate	Validate Open Ingestion Standard (OIS) schemas offline.
oc config	set, get, context	Manage active server URL contexts and credentials.

For complete flag details, environment variables, and GitHub Actions workflow integration examples, visit the OpenCrawling CLI Wiki Documentation.

---

🤖 AI-Powered Observability & Log Analysis (AIOps)

OpenCrawling incorporates AI-Powered Observability (AIOps) to automatically translate complex OpenTelemetry (OTel) distributed traces, Micrometer performance metrics, and Virtual Thread execution stack traces into plain, human-readable Root Cause Analysis (RCA) reports.

Key Capabilities

• Instant Root Cause Analysis (RCA): Click "Diagnose with AI" next to any pipeline job in oc-admin-ui to analyze OTel spans, identify exact failure points (e.g. database insertion timeouts, network latency), and receive actionable fix recommendations.
• Correlated OpenTelemetry Spans: All major pipeline stages (Scanning, Extracting, Chunking, Embedding, Indexing) record correlated OTel spans with timing breakdowns.
• OTel Trace Timeline Visualizer: Inspect trace spans and child concurrent virtual thread spans in real-time, detailing component names, latency bars, statuses, and custom span parameters.
• Error & Exception Log Analyzer: Directly parses and displays full exception stack traces inside the UI, making troubleshooting immediate.
• Toggleable Telemetry: Toggle observability dynamically using environment flags:
  • OPENCRAWLING_OBSERVABILITY_ENABLED (true/false) to toggle tracing.
  • OPENCRAWLING_OBSERVABILITY_SAMPLING_PROBABILITY (0.0 to 1.0) to configure sampling rates.
• System MCP Tools: Admin Copilot exposes native Model Context Protocol (MCP) tools for LLM diagnostic queries:
  • fetch_job_traces(jobId): Retrieves correlated OTel spans and timing breakdowns per pipeline stage.
  • get_error_logs(jobId, timeframe): Fetches failure logs and exception stack traces.
  • query_throughput_metrics(connectorId): Queries throughput rates (docs/sec), P95 latency, and active virtual threads.

---

⚡ gRPC High-Performance Internal Transport

OpenCrawling supports gRPC as an optional, high-performance binary transport protocol for internal component payload communication across microservices (e.g. oc-repository-connectors, oc-core, oc-worker, oc-output-connectors). By default, transport operates in AUTO mode.

Key Features

• Up to 50% Payload Size Reduction: Protobuf binary serialization compresses document metadata and chunk payload transport compared to verbose JSON strings.
• Protocol Modes (AUTO, GRPC, REST):
  • AUTO (Default / Recommended): Attempts high-speed gRPC binary streaming over port 9095. If unreachable or timing out, automatically falls back to HTTP/REST.
  • GRPC (Strict): Enforces Protobuf gRPC streaming across all processing nodes.
  • REST (Standard): Disables gRPC server and uses standard HTTP/REST JSON endpoints.
• TLS / mTLS Encryption: Supports certificate-based transport security for enterprise compliance (tlsEnabled=true).
• Live Admin UI Diagnostic Probes: Test gRPC channel connectivity and latency directly from the Internal Communication & Transport Settings Panel in oc-admin-ui or via POST /api/v1/admin/settings/transport/test-grpc.

// Protobuf Payload Contract (oc-grpc-api)
service InternalPayloadService {
  rpc StreamDocumentPayloads (stream DocumentPayloadRequest) returns (stream PayloadIngestionResponse);
  rpc SendDocumentPayload (DocumentPayloadRequest) returns (PayloadIngestionResponse);
  rpc PingTransport (PingRequest) returns (PingResponse);
}

Tip

See gRPC Internal Transport in the Wiki for full Protobuf schema definitions, YAML properties, and environment variables.

---

📖 OpenAPI 3.1 & Interactive Swagger UI

OpenCrawling provides native, platform-wide OpenAPI 3.1 specification generation and an interactive Swagger UI explorer bundled directly within oc-runtime.

Endpoints

• Interactive Swagger UI: http://localhost:8080/swagger-ui.html
• OpenAPI 3.1 JSON Specification: http://localhost:8080/v3/api-docs

Core Capabilities

• Strict OpenAPI 3.1 Compatibility: Leveraging springdoc-openapi 3.0+ optimized for Java 25 and Spring Boot 4.
• Complete Domain Coverage: Comprehensive @Operation, @ApiResponse, and @Parameter annotations across all 10 logical platform domains:
  • Pipeline & Jobs: Lifecycle orchestration (POST /api/jobs/{id}/start, stop, pause, resume, pagination, status).
  • Connectors: Dynamic repository, transformation, and output connector discovery and connection checks.
  • OIS Direct Ingestion: High-throughput, token-secured direct document ingestion (POST /api/ois/ingest).
  • Reports & Lakehouse Audit: Job summaries, Parquet Lakehouse audit trails, DuckDB/Iceberg query executions.
  • Observability & AIOps: Live pipeline telemetry, trace timeline visualizer, metrics, and error logs.
  • System & Settings: System settings and runtime configuration management.
  • Transport Settings: High-performance internal transport configuration and live gRPC diagnostic ping probes.
  • ManifoldCF Migration: Automated Apache ManifoldCF job and connector definition conversion.
  • Vespa Insights: Vespa cluster connectivity diagnostics, document counts, BM25 text queries, and schema deployments.
  • Auto-Narrativization Copilot: AI-driven Mustache narrative template generation from structured schema definitions.
• Security Schemes: Includes explicit documentation for OisTokenAuth HTTP Bearer token authentication on secure endpoints.
• Client SDK Generation: The generated /v3/api-docs spec can be directly consumed by OpenAPI Generator to synthesize typed clients in TypeScript, Python, Go, or Rust.

---

📝 Auto-Narrativization Copilot

OpenCrawling introduces an Auto-Narrativization Copilot — an AI-powered pipeline that automatically translates structured dataset schemas (field names, types, and descriptions) into human-readable Mustache narrative templates and corresponding mock datasets, ready to be used in document transformation pipelines.

How It Works

Schema Context (connectorType + fields)
        │
        ▼
 TemplateGenerationCopilot
  ├── Calls Ollama / OpenAI via Spring AI (if available)
  └── Falls back to deterministic template generation (offline-safe)
        │
        ▼
 TemplateCopilotResponse
  ├── template  → Mustache template string  e.g. "Record {{id}} processed {{amount}} in {{region}} on {{timestamp}}"
  └── mockData  → JSON map with field-typed sample values
        │
        ▼
 MustacheTransformationConnector  (oc-core)
  └── Renders the template against a live RepositoryDocument's metadata
        │
        ▼
  Enriched narrative text stored in the document content stream

Key Components

Component	Module	Role
MustacheTransformationConnector	oc-core	Renders Mustache templates against RepositoryDocument metadata using JMustache 1.16
ConnectorSchema / SchemaField	oc-core	SPI record that exposes a connector's field schema (name, type, description)
RepositoryConnector.getSchema()	oc-core	Default SPI method returning ConnectorSchema.UNKNOWN; connectors may override
IcebergRepositoryConnector.getSchema()	oc-iceberg-repository-connector	Returns the live Apache Iceberg table schema as a ConnectorSchema
TemplateGenerationCopilot	oc-runtime	Spring AI client that calls the configured LLM (Ollama default) or falls back to deterministic generation
NarrativizationCopilotController	oc-runtime	REST controller exposing POST /api/transformation/copilot/generate
OpenCrawlingClient	oc-java-client-sdk	Strongly typed Java Client SDK & Spring Boot starter for jobs, connectors, AIOps diagnostics, and copilot

REST API

POST /api/transformation/copilot/generate
Content-Type: application/json

{
  "connectorType": "iceberg",
  "fields": [
    { "name": "id",        "type": "STRING", "description": "Primary record identifier" },
    { "name": "amount",    "type": "DOUBLE", "description": "Transaction monetary value" },
    { "name": "region",    "type": "STRING", "description": "Geographical region code" },
    { "name": "timestamp", "type": "DATE",   "description": "Transaction timestamp" }
  ]
}

Response:

{
  "template": "Record {{id}} processed {{amount}} in {{region}} on {{timestamp}}.",
  "mockData": {
    "id":        "Sample id",
    "amount":    123.45,
    "region":    "Sample region",
    "timestamp": "2026-01-01"
  }
}

Per-Job Configuration

Narrativization is configured on a per-job basis. Each ingestion job can define its own Mustache template or disable narrativization entirely:

{
  "id": "job-101",
  "name": "Iceberg_Sales_EU",
  "repositoryConnector": "Iceberg_Local",
  "outputConnector": "PGVector_Output",
  "path": "sales_db.eu_transactions",
  "narrativization": {
    "enabled": true,
    "template": "Transaction {{id}} processed amount €{{amount}} in region {{region}} at {{timestamp}}.",
    "connectorType": "iceberg"
  }
}

When enabled: true, JobOrchestrator applies the MustacheTransformationConnector to render the document's metadata before publishing to Kafka.

AI Engine Configuration

The Copilot defaults to Ollama using llama3.2 as its chat model. This is configured via Spring properties:

spring:
  ai:
    copilot:
      engine: ollama   # default — override with 'openai' to use OpenAI
    ollama:
      base-url: http://127.0.0.1:11434
      chat:
        options:
          model: ${SPRING_AI_OLLAMA_CHAT_MODEL:llama3.2}
          keep_alive: 1m

When Ollama is unavailable or the chat model is not loaded, the service automatically falls back to a deterministic template generator that produces valid Mustache templates and typed mock values from the schema metadata — ensuring the API is always reliable.

Integration Test

A dedicated layered integration test script is provided under scripts/:

# Layers 1–3: Mustache engine, Schema SPI, Copilot REST API (no infrastructure required)
./scripts/test-narrativization.sh

# Layer 4: End-to-end curl test against a running OpenCrawling instance
./scripts/test-narrativization.sh --e2e

# Point to a custom host
./scripts/test-narrativization.sh --e2e --url http://staging.host:8080

The script covers:

Layer	What is tested
Layer 1	MustacheTransformationConnector JUnit test + JMustache JAR presence
Layer 2	ConnectorSchema SPI compilation + IcebergRepositoryConnector.getSchema() override
Layer 3	NarrativizationCopilotIT (Mockito), Ollama default config assertion, DTO shape checks
Layer 4	Live HTTP POST /api/transformation/copilot/generate + response JSON assertions

---

📦 Custom Connectors: Maven Archetypes Suite

OpenCrawling provides an official suite of Maven Archetypes under opencrawling-connector-archetypes allowing developers and ecosystem partners to instantly scaffold ready-to-build, standardized custom connectors with a single mvn archetype:generate command.

Available Archetypes

Archetype Artifact ID	Maven Central	Connector Type	Description & SPI Interface
opencrawling-archetype-repository-connector		Repository Connector	Ingestion sources (CMS, REST APIs, Databases, File Systems). Implements RepositoryConnector (Flux<RepositoryDocument>).
opencrawling-archetype-output-connector		Output Connector	Ingestion destinations (Vector DBs, OpenSearch, Elasticsearch, S3). Implements OutputConnector (Mono<Void>).
opencrawling-archetype-transformation-connector		Transformation Connector	Data processing, chunking, and AI enrichment. Implements TransformationConnector (Flux<RepositoryDocument>).

Quickstart CLI Commands

# Generate a Repository Connector (Ingestion Source)
mvn archetype:generate \
  -DarchetypeGroupId=org.opencrawling.archetypes \
  -DarchetypeArtifactId=opencrawling-archetype-repository-connector \
  -DarchetypeVersion=1.0.4-SNAPSHOT \
  -DgroupId=org.opencrawling.connectors \
  -DartifactId=opencrawling-repository-sample \
  -Dversion=1.0.4-SNAPSHOT \
  -DconnectorName=SampleRepositoryConnector

# Generate an Output Connector (Vector Store / Destination)
mvn archetype:generate \
  -DarchetypeGroupId=org.opencrawling.archetypes \
  -DarchetypeArtifactId=opencrawling-archetype-output-connector \
  -DarchetypeVersion=1.0.4-SNAPSHOT \
  -DgroupId=org.opencrawling.connectors \
  -DartifactId=opencrawling-output-sample \
  -Dversion=1.0.4-SNAPSHOT \
  -DconnectorName=SampleOutputConnector

# Generate a Transformation Connector (Data Processing / Enrichment)
mvn archetype:generate \
  -DarchetypeGroupId=org.opencrawling.archetypes \
  -DarchetypeArtifactId=opencrawling-archetype-transformation-connector \
  -DarchetypeVersion=1.0.4-SNAPSHOT \
  -DgroupId=org.opencrawling.connectors \
  -DartifactId=opencrawling-transformation-sample \
  -Dversion=1.0.4-SNAPSHOT \
  -DconnectorName=SampleTransformationConnector

Docker Compose Overlay & Integration Testing

Every generated archetype comes pre-configured with:

• Unit & Integration Test Suite: Uses JUnit 5, Reactor StepVerifier, and maven-failsafe-plugin (*IT.java execution via mvn verify).
• Official OpenCrawling Docker Compose Distribution (docker/docker-compose.dist.yml): Launches OpenCrawling backend, Admin UI (http://localhost:3000), Postgres + pgvector, Redis Stack, and Kafka.
• Docker Compose Overlay (docker/docker-compose.override.yml): Overlays the custom compiled connector JAR into the running OpenCrawling container volume (/app/plugins/).

# Build custom connector JAR
mvn clean package

# Start OpenCrawling Docker environment with custom connector overlay
docker compose -f docker/docker-compose.dist.yml -f docker/docker-compose.override.yml up -d

# Interactively configure, run, and monitor jobs using your connector in Admin UI:
# Open http://localhost:3000 in your browser

# Execute unit and integration tests
mvn verify

# Tear down Docker environment
docker compose -f docker/docker-compose.dist.yml -f docker/docker-compose.override.yml down

---

Core Technologies

• Java 25 Preview Features: Structured Concurrency, Virtual Threads, and Pattern Matching.
• Spring Boot & Spring AI: High-performance backend orchestrating ingestion jobs and MCP Tool calling.
• gRPC High-Performance Binary Transport: Protobuf-backed binary payload channel (Port 9095) with automatic HTTP/REST fallback, TLS/mTLS encryption, and live Admin UI diagnostic ping testing.
• OpenTelemetry & Micrometer AIOps: Automated Root Cause Analysis (RCA) and correlated distributed span telemetry.
• Model Context Protocol (MCP): System tools exposing vector search and OTel telemetry to LLMs.
• OpenFGA (optional): Relationship-based document authorization; one access gate for every store, with shadow and intersection modes for a safe rollout.
• Apache Kafka: Decoupled, event-driven document processing using the Claim Check Pattern.
• Apache Ozone 2.2.0: High-performance distributed object store offloading large document payloads with dual client strategy:
  • Native Ozone Client (ofs / o3fs): Direct gRPC/RPC transport to DataNodes & Ozone Manager (OM) for maximum throughput.
  • S3 Gateway Client (s3g): Standard AWS S3 SDK integration hitting Ozone's S3 Gateway endpoint for maximum cloud versatility.
• Repository Connectors: Multi-source connectors for Filesystem, Alfresco Content Services (ACS), Alfresco Process Services (APS), OASIS CMIS (1.0/1.1), Doxis 4 (CSB REST), SharePoint Online (Microsoft Graph), M-Files Vault (MFWS REST), IBM FileNet P8 (Content Services GraphQL), OpenText Documentum (REST Services), Hyland OpenArch, Apache Iceberg, Flowable BPMN, Camunda, Apache StormCrawler, and gRPC Push Ingestion Gateway.
• Hyland OpenArch & Parquet Vector Bridging: Native bidirectional integration supporting paged HXQL harvesting, binary blob offloading via Java 25 Virtual Threads, Parquet vector serialization (embedding.avsc), Zero-Trust ACL preservation, and MCP search/fetch tools (openarch_search, openarch_fetch_document).
• pgvector: High-dimensional vector similarity search in PostgreSQL.
• Milvus: High-performance, distributed vector database for large-scale enterprise vector indexing.
• Qdrant: Rust-based vector search engine with payload-indexed ACL pre-filtering and optional scalar/binary quantization.
• Vespa: Hybrid ANN vector search and BM25 lexical ranking engine, combining nearestNeighbor and full-text queries in a single rank profile for enterprise-scale RAG retrieval.
• Redis Stack: Lightweight caching and session management.
• Ollama & OpenAI: Dynamic embedding generation via local and cloud-based AI engines.
• Vite + React + TailwindCSS: Modern frontend administration dashboard with interactive AIOps diagnostic panels.

---

Getting Started

Prerequisites

Ensure you have the following installed on your machine:

• JDK 25 (Ensure JAVA_HOME points to your JDK 25 directory)
• Maven 3.9+
• Docker & Docker Compose
• Node.js 18+ & npm (for the UI)

---

Quick Start with run.sh / run.bat

For the fastest local developer experience on macOS / Linux (./run.sh) and Windows CMD (run.bat), use the root developer CLI convenience wrappers. They encapsulate BuildKit ~/.m2 cache mounts, containerized Maven builds (oc-maven-builder), Docker Compose stack lifecycles, testing, and Admin UI commands:

# macOS / Linux
./run.sh help              # View all available commands
./run.sh build             # Compile all modules via oc-maven-builder using host ~/.m2 cache
./run.sh decoupled:up      # Build & start the decoupled stack with BuildKit ~/.m2 cache mounts
./run.sh apps:up           # Build & start the monolith apps stack (backend, embedding, UI)
./run.sh status            # Inspect running containers and port bindings
./run.sh logs oc-crawler   # Stream container logs
./run.sh ui:dev            # Start the React Admin UI Vite dev server on :5173
./run.sh down              # Stop all active local stacks

REM Windows Command Prompt (CMD)
run.bat help
run.bat build
run.bat decoupled:up
run.bat apps:up
run.bat status
run.bat down

Category	Command (./run.sh or run.bat)	Description
Build & Maven	build / mvn:build	Compile all modules via containerized oc-maven-builder (-DskipTests)
	build:offline	Fast offline build via oc-maven-builder (-o -DskipTests)
	build:host / build:native	Build using host JDK 25 & Maven (mvn clean package -DskipTests)
	build:clean	Full clean build with unit tests via oc-maven-builder
	mvn <args...>	Run custom arbitrary Maven goals inside oc-maven-builder
Docker Stacks	decoupled:up / decoupled:local	Start standalone decoupled stack with local BuildKit cache mounts
	decoupled:override	Start decoupled stack via docker-compose-decoupled.local.yml overlay
	decoupled:down	Stop decoupled local stack
	apps:up / apps:local	Start standalone apps stack (backend, embedding, UI) with local Dockerfiles
	apps:override	Start apps stack via docker-compose-apps.local.yml overlay
	apps:down	Stop apps local stack
	infra:up / infra:down	Start or stop core infrastructure only (PostgreSQL/pgvector, Redis, Ollama, Kafka)
	down / stop	Stop all active local OpenCrawling stacks
Productivity	status / ps	Show status and port bindings of running local containers
	logs [service...]	Follow container logs (optionally filtered by service)
	test:unit [args...]	Run unit tests via oc-maven-builder
	test:it [filter...]	Run integration tests with local BuildKit cache overlay
	test:local-dev	Run dedicated local Docker build acceleration test suite
	ui:dev / ui:build	Start Vite dev server (:5173) or build React Admin UI bundle
	license:format / format	Apply Apache 2.0 license headers across project files
	clean [--prune]	Clean Maven target/ folders (and optionally prune Docker build cache)

---

Step-by-Step Setup

1. Start Infrastructure (Docker)

Spin up the database, cache, message broker, and AI engine. Run from the project root:

docker compose up -d

Services started:

• PostgreSQL (Port 5432): For job metadata, schema migrations, and pgvector storage.
• Redis (Port 6379 / Insight Port 8001): For caching and session management.
• Ollama (Port 11434): For local embeddings.
• Apache Kafka (Port 9092): KRaft-mode broker for decoupled, event-driven document processing.

2. Pull the Embedding Models (Ollama)

OpenCrawling supports configuring different embedding models on a per-job basis and automatically routes them to corresponding PgVector tables. To use the available options, make sure to pull the models you plan to utilize:

• mxbai-embed-large (1024-dim, default):

  docker exec -it ollama ollama pull mxbai-embed-large

• nomic-embed-text (768-dim):

  docker exec -it ollama ollama pull nomic-embed-text

• all-minilm (384-dim):

  docker exec -it ollama ollama pull all-minilm

(Ollama will download the requested models in the background. Once pulled, OpenCrawling will automatically route them to vector_store_1024, vector_store_768, or vector_store_384 respectively).

---

Option A: Run OpenCrawling in Docker Containers (Recommended)

To build and run the OpenCrawling backend runtime, the dynamic embedding microservice, and the administration UI as containerized services, run:

1. Build the images:

   docker compose -f docker-compose-apps.yml build

2. Start the applications:

   docker compose -f docker-compose-apps.yml up -d

• Backend Service: Access the backend runtime and integrated static resources at http://localhost:8080.
• Embedding Service: The dynamic microservice processes embeddings at http://localhost:8082.
• Frontend Service: Access the standalone React Administration Console at http://localhost:3000.

---

Option A.1.b: Run Pre-built Stack from Docker Hub

To run the standard multi-tier application stack using pre-built images from Docker Hub (without building them from source):

1. Start backing services:

   docker compose up -d

2. Start application services:

   docker compose -f docker-compose-apps-dockerhub.yml up -d

---

Option A.2: Decoupled Multi-Service Deployment (Decoupled Microservices)

To run each microservice component (Repository Crawler, Ingestion Consumer, Embedding Consumer, Vector Store Writer, Secure MCP Server, and Admin UI) as a completely separate containerized process communicating over Kafka:

1. Build the decoupled service images:

   docker compose -f docker-compose-decoupled.yml build

2. Start the complete decoupled pipeline:

   docker compose -f docker-compose-decoupled.yml up -d

This spins up the database/event-stream dependencies alongside five decoupled OpenCrawling service containers. You can view logs, scale individual workers (e.g. docker compose -f docker-compose-decoupled.yml scale oc-embedding-service=3), and monitor the decoupled pipeline.

• React Admin UI Console: Access the administration dashboard at http://localhost:3000.
• Secure MCP Server: Connect your AI Client / IDE directly to http://localhost:8080 over SSE.

---

Option A.2.b: Run Decoupled Stack from Docker Hub

To run the complete decoupled pipeline using pre-built images published to Docker Hub:

1. Start the decoupled pipeline:

   docker compose -f docker-compose-decoupled-dockerhub.yml up -d

This spins up the entire backing infrastructure (Postgres, Redis, Kafka, Ollama, OpenTelemetry stack) and all five decoupled OpenCrawling services instantly using Docker Hub images.

---

Option A.2.c: Fast Local Development Builds & oc-maven-builder (Local Maven Cache ~/.m2)

In CI workflows (GitHub Actions), container images are built from scratch to guarantee clean, hermetic builds. However, for local development, downloading all dependencies repeatedly across multi-service builds (oc-crawler, oc-ingestion-consumer, oc-embedding-consumer, oc-writer-consumer, oc-mcp-server) creates significant network overhead.

To improve the local development experience, OpenCrawling provides dedicated local configurations that preserve the CI Dockerfiles untouched while leveraging the developer's local Maven repository cache (<USER_HOME>/.m2).

1. The oc-maven-builder Service (Host ~/.m2 Volume Mount)

The oc-maven-builder service (defined in docker-compose-maven-builder.yml and included across local compose files) runs the official maven:3-eclipse-temurin-25 container with two crucial volume mounts:

• Host Maven Cache Mount (${HOME}/.m2:/root/.m2): Any dependency downloaded by Maven is stored directly on your host machine's ~/.m2 folder. Existing dependencies on your machine are reused immediately with zero download time.
• Workspace Mount (.:/app): Compiles and outputs JAR artifacts (target/*.jar) directly into your host workspace.
• Zero-Host-Prerequisites: Developers do not need JDK 25 or Maven installed on their host machine; builds execute inside the exact containerized Java 25 environment.
• Docker Compose Profile (profiles: ["build"]): It is marked with profiles: ["build"], so it never starts as an unwanted background daemon during docker compose up -d. It only runs on-demand when called with run --rm.

2. Usage with Decoupled Pipeline Stack

To compile and launch the decoupled microservices:

# 1. Compile all project modules using host ~/.m2
docker compose -f docker-compose-decoupled-local.yml run --rm oc-maven-builder

# 2. Start the decoupled microservices
docker compose -f docker-compose-decoupled-local.yml up -d

You can also use the local override on top of docker-compose-decoupled.yml:

docker compose -f docker-compose-decoupled.yml -f docker-compose-decoupled.local.yml up -d --build

3. Usage with Monolith Apps Stack (Backend + UI)

# 1. Compile modules using host ~/.m2
docker compose -f docker-compose-apps-local.yml run --rm oc-maven-builder

# 2. Start the backend, embedding service, and React UI
docker compose -f docker-compose-apps-local.yml up -d

4. Usage with Any Specialized Connector Stack

Because docker-compose-maven-builder.yml is located at the root of the repository, it can be used to compile artifacts before launching any specialized connector compose stack (Vespa, Solr, Luxir, Milvus, Qdrant, SeaTunnel, Ozone):

# 1. Compile project modules
docker compose -f docker-compose-maven-builder.yml run --rm oc-maven-builder

# 2. Launch the target connector stack (e.g. Vespa)
docker compose -f oc-vespa-output-connector/docker/docker-compose-decoupled-with-vespa.yml up -d

5. Running Custom Maven Goals and Module Testing

You can pass arbitrary Maven commands to oc-maven-builder:

# Run unit tests for oc-runtime only
docker compose -f docker-compose-maven-builder.yml run --rm oc-maven-builder mvn test -pl oc-runtime

# Fast offline build (skipping remote dependency repository lookups)
docker compose -f docker-compose-maven-builder.yml run --rm oc-maven-builder mvn clean package -o -DskipTests

# Inspect connector dependency tree
docker compose -f docker-compose-maven-builder.yml run --rm oc-maven-builder mvn dependency:tree -pl oc-vector-output-connector

6. Transparent Fast Builds for Existing Integration Test Scripts

To run integration test scripts under scripts/ using the fast local cache Dockerfiles without modifying any script file:

export COMPOSE_FILE="docker-compose-decoupled.yml:docker-compose-decoupled.local.yml"
./scripts/test-ois-ingest-decoupled.sh

---

Option A.3: Quick Start with Released Containers (Pre-built Release Distribution from GHCR)

To run the complete decoupled pipeline using the official pre-built release containers from the GitHub Container Registry (without building the services locally):

1. Start the release pipeline:

   docker compose -f docker-compose-decoupled-dist.yml up -d

This pulls the official ghcr.io/opencrawling/... images directly, allowing you to spin up the entire architecture (Crawler, Ingestion, Embedding Service, Writer, MCP Server, and Admin UI) instantly.

---

Option A.4: Apache Ozone Claim Check Strategy Configuration

OpenCrawling supports offloading raw payload streams to Apache Ozone via Spring environment properties or the Admin UI:

opencrawling:
  claim-check:
    store: ozone                      # Options: ozone, local
    cleanup-on-consume: true         # Immediate explicit deletion post-ACK
    lifecycle:
      enable-background-gc: true      # Background runner for orphaned payloads
      ttl-hours: 24                   # Retention window before GC purging
      gc-cron: "0 0 * * * *"          # Hourly GC cron schedule
    ozone:
      client-type: NATIVE             # Options: NATIVE (ofs/o3fs RPC) or S3 (s3g/HTTP)
      om-host: "localhost"
      om-port: 9862
      volume: "s3v"
      bucket: "claims"
      s3-endpoint: "http://localhost:9878"
      access-key: "ozone"
      secret-key: "ozone-secret"

• NATIVE (Default / High Performance): Direct gRPC/RPC communication with DataNodes & Ozone Manager (ofs://volume/bucket/key), providing sub-millisecond object offloading.
• S3 (Cloud Versatility): Uses the AWS S3 SDK to target Ozone's S3 Gateway (s3://bucket/key).
• cleanup-on-consume: true: Triggers immediate explicit deletion of offloaded payloads as soon as Kafka chunk extraction receives a post-ingestion ACK.
• lifecycle.enable-background-gc: true: Runs a background ClaimCheckGarbageCollector on the configured cron schedule (gc-cron) to clean up orphaned claim-check objects older than ttl-hours caused by node crashes or pipeline failures.

---

Option A.4: Decoupled Milvus-Based Deployment (Standalone + etcd + MinIO)

To run the complete decoupled pipeline configured to use Milvus instead of PostgreSQL/pgvector:

1. Build the Milvus decoupled stack:

   docker compose -f oc-milvus-output-connector/docker/docker-compose-decoupled-with-milvus.yml build

2. Start the Milvus decoupled pipeline:

   docker compose -f oc-milvus-output-connector/docker/docker-compose-decoupled-with-milvus.yml up -d

This starts the etcd, MinIO, and Milvus Standalone infrastructure alongside the decoupled OpenCrawling services.

---

Option A.5: Decoupled Qdrant-Based Deployment

To run the complete decoupled pipeline configured to use Qdrant instead of PostgreSQL/pgvector:

1. Build the Qdrant decoupled stack:

   docker compose -f oc-qdrant-output-connector/docker/docker-compose-decoupled-with-qdrant.yml build

2. Start the Qdrant decoupled pipeline:

   docker compose -f oc-qdrant-output-connector/docker/docker-compose-decoupled-with-qdrant.yml up -d

This starts a standalone Qdrant instance alongside the decoupled OpenCrawling services. Key configuration properties (see spring.opencrawling.output.qdrant.*): host, port (gRPC, default 6334), api-key (for Qdrant Cloud), collection-name, dimensions, distance (COSINE | DOT | EUCLID), quantization (NONE | SCALAR_INT8 | BINARY), use-tls, and batch-size. ACL pre-filtering works the same way as with pgvector/Milvus: security_allowed_read/security_denied_read keyword payload indexes are auto-provisioned on the collection at startup.

---

Option A.6: Decoupled Vespa-Based Deployment

To run the complete decoupled pipeline configured to use Vespa instead of PostgreSQL/pgvector:

1. Build the Vespa decoupled stack:

   docker compose -f oc-vespa-output-connector/docker/docker-compose-decoupled-with-vespa.yml build

2. Start the Vespa decoupled pipeline:

   docker compose -f oc-vespa-output-connector/docker/docker-compose-decoupled-with-vespa.yml up -d

This starts a standalone Vespa instance, deploys its schema application package (oc-vespa-output-connector/vespa-app/) via a one-shot container, and boots the decoupled OpenCrawling services alongside it. Key configuration properties (see spring.opencrawling.output.vespa.*): endpoint, namespace, document-type (fallback only, see below), dimensions, timeout-seconds, and tls-enabled/tls-certificate/tls-private-key/tls-ca-certificates (Vespa Cloud mTLS). Each chunk is routed dynamically to a dimension-specific document type (opencrawling_chunk_384/768/1024) based on its actual embedding length, so all-minilm, nomic-embed-text, and mxbai-embed-large can be fed side by side with no redeploy; document-type/dimensions are only used as the fallback for any other embedding size. Unlike pgvector/Milvus/Qdrant, the connector does not auto-provision the schema on startup; all four document types are deployed once via the Vespa config server's deploy REST API, and ACL pre-filtering is enforced through security_allowed_read/security_denied_read fields declared in that schema.

---

Option A.7: Decoupled Apache Solr-Based Deployment

To run the complete decoupled pipeline configured to use Apache Solr instead of PostgreSQL/pgvector:

1. Build the Solr decoupled stack:

   docker compose -f oc-solr-output-connector/docker/docker-compose-decoupled-with-solr.yml build

2. Start the Solr decoupled pipeline:

   docker compose -f oc-solr-output-connector/docker/docker-compose-decoupled-with-solr.yml up -d

---

Option A.8: Decoupled Luxir-Based Deployment

To run the complete decoupled pipeline configured to use Luxir instead of PostgreSQL/pgvector:

1. Build the Luxir decoupled stack:

   docker compose -f oc-luxir-output-connector/docker/docker-compose-decoupled-with-luxir.yml build

2. Start the Luxir decoupled pipeline:

   docker compose -f oc-luxir-output-connector/docker/docker-compose-decoupled-with-luxir.yml up -d

This starts a standalone Luxir search engine instance alongside the decoupled OpenCrawling services. Key configuration properties (see spring.opencrawling.output.luxir.*): endpoint, collection, vector-field (embedding_v), dimensions (384 / 768 / 1024), similarity (cosine), and auto-commit (true). Luxir automatically handles dense vector kNN search, dynamic typing schemas (_t, _s, _ss, _v), OIS v1.1 tombstone deletions via delete_ids, and zero-trust ACL token filtering.

Option A.8.b: Doxis 4 Repository Crawling

To crawl a Doxis 4 DMS repository, select the Doxis repository connector and point it at a Doxis CSB:

Prerequisites: you need a valid Doxis licence that allows API (technical-user) sessions, and a reachable Doxis 4 CSB 14.4+ with its REST API enabled. You also need a technical user and role with read rights on the repository, its classes, e-files and users/groups/roles, and an existing DMS repository. OpenCrawling doesn't include Doxis or any Doxis licence. See the full prerequisites before configuring a crawl.

SPRING_OPENCRAWLING_REPOSITORY_CONNECTOR_TYPE=doxis \
SPRING_OPENCRAWLING_CONNECTOR_TYPE=doxis \
SPRING_OPENCRAWLING_CONNECTOR_DOXIS_URL=http://<csb-host>:8080/restws/publicws/rest/api/v1 \
SPRING_OPENCRAWLING_CONNECTOR_DOXIS_CUSTOMER_NAME=<customer> \
SPRING_OPENCRAWLING_CONNECTOR_DOXIS_USERNAME=<user> \
SPRING_OPENCRAWLING_CONNECTOR_DOXIS_PASSWORD=<password> \
SPRING_OPENCRAWLING_CONNECTOR_DOXIS_REPOSITORY_ID=<repository> \
SPRING_OPENCRAWLING_CONNECTOR_DOXIS_SEARCH_QUERY="OBJECTNAME LIKE 'Contract*'" \
mvn spring-boot:run -pl oc-runtime -Dspring-boot.run.profiles=dev

DoxisRepositoryConnector reads documents through the Doxis CSB REST API.

• What it crawls: either one CQL search (crawl-mode: search) or an e-file (Akte) and its folder nodes (crawl-mode: folder).
• What each document carries: its descriptors, the content of its default representation, and the document and e-file ACLs as OIS permissions.
• Deletes: logically removed documents become DELETE tombstones.
• Sessions: a scan uses one CSB session and always logs out.

In the Admin UI it is the Doxis Repository repository connector. See oc-doxis-repository-connector/README.md.

Option A.8.c: M-Files Vault Crawling

To crawl an M-Files vault, select the M-Files repository connector and point it at the M-Files Web Service (MFWS):

Prerequisites: you need a licensed M-Files Server with the M-Files Web Service (REST API) enabled, and an M-Files account with read access to the vault (or an OAuth2 / Entra ID client the vault accepts). OpenCrawling doesn't include M-Files or any M-Files licence. See the full prerequisites before configuring a crawl.

SPRING_OPENCRAWLING_REPOSITORY_CONNECTOR_TYPE=mfiles \
SPRING_OPENCRAWLING_CONNECTOR_TYPE=mfiles \
SPRING_OPENCRAWLING_CONNECTOR_MFILES_URL=https://<mfiles-host>/REST \
SPRING_OPENCRAWLING_CONNECTOR_MFILES_VAULT_GUID="{<vault-guid>}" \
SPRING_OPENCRAWLING_CONNECTOR_MFILES_USERNAME=<user> \
SPRING_OPENCRAWLING_CONNECTOR_MFILES_PASSWORD=<password> \
SPRING_OPENCRAWLING_CONNECTOR_MFILES_CLASS_IDS=14,22 \
mvn spring-boot:run -pl oc-runtime -Dspring-boot.run.profiles=dev

MFilesRepositoryConnector reads objects through the MFWS REST API.

• What it crawls: MFWS searches by object type, class and conditions (crawl-mode: search / all_objects), an M-Files view (view), or the change feed of a Vault Application Framework app (vaf).
• What each document carries: the object's typed property values, its file (one OIS document per file for a multi-file document), and its permissions as OIS permissions, deny by default.
• Deletes: deleted objects become DELETE tombstones; incremental crawls also purge files an object no longer has.
• Sessions: a scan uses one MFWS session and always ends it.

In the Admin UI it is the M-Files Repository repository connector. See oc-mfiles-repository-connector/README.md.

Option A.8.d: SharePoint Online Repository Crawling

To crawl the document libraries of a SharePoint Online site, select the SharePoint repository connector and give it an Entra ID app registration:

Prerequisites: a Microsoft Entra ID app registration with a client secret and the admin-consented Microsoft Graph application permission Sites.Read.All (or Sites.Selected with read access to the site). See the full prerequisites before configuring a crawl.

SPRING_OPENCRAWLING_REPOSITORY_CONNECTOR_TYPE=sharepoint \
SPRING_OPENCRAWLING_CONNECTOR_TYPE=sharepoint \
SPRING_OPENCRAWLING_CONNECTOR_SHAREPOINT_TENANT_ID=<tenant-id> \
SPRING_OPENCRAWLING_CONNECTOR_SHAREPOINT_CLIENT_ID=<client-id> \
SPRING_OPENCRAWLING_CONNECTOR_SHAREPOINT_CLIENT_SECRET=<client-secret> \
SPRING_OPENCRAWLING_CONNECTOR_SHAREPOINT_SITE_URL=https://<tenant>.sharepoint.com/sites/<site> \
mvn spring-boot:run -pl oc-runtime -Dspring-boot.run.profiles=dev

SharePointRepositoryConnector reads files through Microsoft Graph v1.0.

• What it crawls: every document library of the site, or the ones named in drive-names, through Graph delta queries (crawl-mode: delta, incremental) or a folder traversal (crawl-mode: tree).
• What each file carries: its list item columns, its content, and its Graph permissions as OIS permissions. Files without readers stay closed.
• Deletes: deleted files and folders become DELETE tombstones.
• Permission-only changes: a periodic tree crawl picks them up.

In the Admin UI it is the Microsoft SharePoint (Microsoft Graph API) repository connector. The Graph mock emulates a tenant for local testing. See oc-sharepoint-repository-connector/README.md.

Option A.8.e: IBM FileNet P8 Crawling

To crawl an object store of an IBM FileNet P8 Content Platform Engine, select the FileNet repository connector and point it at the Content Services GraphQL API:

Prerequisites: you need a licensed IBM FileNet P8 installation with the Content Services GraphQL API deployed, and an account with read access to the object store (Basic, OAuth2 / Cloud Pak IAM, a bearer token or an LTPA token). OpenCrawling doesn't include IBM FileNet or any IBM licence. See the full prerequisites before configuring a crawl.

SPRING_OPENCRAWLING_REPOSITORY_CONNECTOR_TYPE=filenet \
SPRING_OPENCRAWLING_CONNECTOR_TYPE=filenet \
SPRING_OPENCRAWLING_CONNECTOR_FILENET_URL=https://<cpe-host>/content-services-graphql/graphql \
SPRING_OPENCRAWLING_CONNECTOR_FILENET_OBJECT_STORE=OS1 \
SPRING_OPENCRAWLING_CONNECTOR_FILENET_USERNAME=<user> \
SPRING_OPENCRAWLING_CONNECTOR_FILENET_PASSWORD=<password> \
SPRING_OPENCRAWLING_CONNECTOR_FILENET_DOCUMENT_CLASSES=LoanApplicationDocument \
mvn spring-boot:run -pl oc-runtime -Dspring-boot.run.profiles=dev

FileNetRepositoryConnector reads documents through the Content Services GraphQL API.

• What it crawls: documents of the configured classes (subclasses included) matching an optional CE SQL condition (crawl-mode: query), or below a folder (folder); the current, the latest released, or every version.
• What each document carries: its class properties, every content element as its own OIS document, the folders it is filed in, and its effective access permissions as OIS permissions, deny by default (a deny of view content alone is a deny).
• Incremental crawls and deletes: a DateLastModified watermark skips unchanged versions; deleted documents are found by reconciling the current ids (no auditing needed) or from the DeletionEvent audit log, and become DELETE tombstones.

In the Admin UI it is the IBM FileNet P8 (Content Platform Engine) repository connector. See oc-filenet-repository-connector/README.md.

Option A.8.f: OpenText Documentum Crawling

To crawl a repository of an OpenText Documentum Content Server, select the Documentum repository connector and point it at Documentum REST Services:

Prerequisites: you need a licensed OpenText Documentum installation with Documentum REST Services deployed, and an account that can read the documents to crawl (Basic with a password or login ticket, OAuth2 / OTDS, or a bearer token). OpenCrawling doesn't include OpenText Documentum or any OpenText licence. See the full prerequisites before configuring a crawl.

SPRING_OPENCRAWLING_REPOSITORY_CONNECTOR_TYPE=documentum \
SPRING_OPENCRAWLING_CONNECTOR_TYPE=documentum \
SPRING_OPENCRAWLING_CONNECTOR_DOCUMENTUM_URL=https://<rest-host>/dctm-rest \
SPRING_OPENCRAWLING_CONNECTOR_DOCUMENTUM_REPOSITORY_NAME=<repository> \
SPRING_OPENCRAWLING_CONNECTOR_DOCUMENTUM_USERNAME=<user> \
SPRING_OPENCRAWLING_CONNECTOR_DOCUMENTUM_PASSWORD=<password> \
SPRING_OPENCRAWLING_CONNECTOR_DOCUMENTUM_OBJECT_TYPES=my_contract \
mvn spring-boot:run -pl oc-runtime -Dspring-boot.run.profiles=dev

DocumentumRepositoryConnector reads documents through Documentum REST Services and DQL.

• What it crawls: objects of the configured types (subtypes included) matching an optional DQL condition (crawl-mode: dql), or below a cabinet or folder (folder); the CURRENT, the latest major, or every version.
• What each document carries: its attributes, including the single and repeating attributes of custom types, its primary content or a preferred rendition (such as PDF), its folder paths, and its permission set as OIS permissions, deny by default (access restrictions become denies; documents with required groups are left out).
• Incremental crawls and deletes: an r_modify_date watermark with keyset paging skips unchanged documents; edited permission sets are detected; destroyed documents are found by reconciling the current ids (no auditing needed) or from the dm_audittrail, and become DELETE tombstones.

In the Admin UI it is the OpenText Documentum (REST Services) repository connector. See oc-documentum-repository-connector/README.md.

Option A.9: Decoupled Apache SeaTunnel-Based Deployment

To run the complete decoupled pipeline configured to fan out document chunks, dense vectors, and security ACLs across 100+ target sinks using Apache SeaTunnel v2.3.13 (Zeta Engine):

1. Start the SeaTunnel Zeta cluster:

   docker compose -f oc-seatunnel-output-connector/docker/docker-compose.yml up -d

This launches a SeaTunnel Zeta cluster alongside OpenCrawling's streaming pipeline. OpenCrawling emits Open Ingestion Standard (OIS) embedded chunks to Apache Kafka (opencrawling-embedded), while SeaTunnelOutputConnector dynamically synthesizes and submits HOCON streaming execution jobs to the SeaTunnel Zeta cluster via its HTTP REST API v2 (http://<master>:8080/submit-job). SeaTunnel fans out document chunks, dense vectors, and security ACLs across target sinks (ClickHouse, Apache Iceberg, Milvus, Qdrant, StarRocks, etc.) with automatic Change Data Capture (RowKind.DELETE) for tombstone purges. Key configuration properties (see spring.opencrawling.output.seatunnel.*): rest-url, job-name, job-mode (STREAMING / BATCH), checkpoint-interval-ms (5000), parallelism (4), kafka-bootstrap-servers, kafka-topic, and target-sinks.

---

Option A.10: Decoupled AWS S3 Migration Deployment

To run the complete decoupled microservice pipeline in Migration Mode streaming original document binaries and companion Zero-Trust OIS JSON sidecars (.ois.json) into Amazon S3 or S3-compatible cloud object stores (MinIO, Ceph, Cloudflare R2):

1. Start the decoupled stack with S3/MinIO:

   docker compose -f oc-s3-output-connector/docker/docker-compose-decoupled-with-s3.yml up -d

This launches the decoupled crawler, ingestion consumer, writer consumer (S3MigrationWriterConsumer), and Secure MCP server with MinIO S3 and Kafka. In Migration Mode, text parsing and vector embeddings are completely bypassed, streaming pristine binaries and writing OIS JSON sidecars with custom HTTP request headers, dynamic template placeholders, and parallel multipart uploads.

---

Running the Decoupled Integration Tests

We provide fully automated end-to-end integration test scripts that build, boot, test, and cleanse the entire decoupled environment:

• PGVector Decoupled Pipeline:

  ./scripts/test-docker-decoupled.sh

  This script tests the decoupled architecture using PostgreSQL and pgvector, verifying database content directly.

• Milvus Decoupled Pipeline:

  ./scripts/test-milvus-decoupled.sh

  This script tests the decoupled architecture using Milvus, querying the Milvus REST API to verify row ingestion and checking Secure MCP Server endpoints.

• Apache Ozone Object Storage Claim Check Pipeline:

  ./scripts/test-ozone-decoupled.sh

  This script tests the decoupled architecture using Apache Ozone 2.2.0 (SCM, OM, Datanode, S3 Gateway) as the Claim Check Object Store for large document payloads.

• AWS S3 Migration Decoupled Pipeline:

  ./scripts/test-s3-migration-decoupled.sh

  This script tests the decoupled architecture in Migration Mode against MinIO / AWS S3, asserting bit-for-bit SHA-256 binary fidelity, companion OIS Zero-Trust JSON sidecars (.ois.json), RAG bypass (0 vector records in pgvector), parallel multipart uploads, custom request headers with dynamic template engine, and OIS tombstone deletions.

• Qdrant Decoupled Pipeline:

  ./scripts/test-qdrant-decoupled.sh

  This script tests the decoupled architecture using Qdrant vector database, querying the Qdrant gRPC/REST API to verify point ingestion and checking Secure MCP Server endpoints.

• Vespa Decoupled Pipeline:

  ./scripts/test-vespa-decoupled.sh

  This script tests the decoupled architecture using Vespa, deploying the schema application package, querying the Vespa Document/Search API to verify chunk ingestion, asserting the oc-runtime Model Insights REST endpoints (health, document counts, BM25 query), and checking Secure MCP Server endpoints.

• Standalone Vespa Connector Smoke Test:

  ./scripts/test-vespa-connector.sh

  This script runs a fast standalone smoke test against a running or temporary Vespa container, deploying vespa-app/, feeding ACL-tagged document chunks at multiple vector dimensions, and verifying dynamic routing (384/768/1024) and search ACL filtering.

• Apache Solr Decoupled Pipeline:

  ./scripts/test-solr-decoupled.sh

  This script tests the decoupled architecture using Apache Solr 10.x / 9.x, verifying HNSW solr.DenseVectorField vector search, schema auto-provisioning, full-text queries, and Secure MCP Server endpoints.

• Luxir Decoupled Pipeline:

  ./scripts/test-luxir-decoupled.sh

  This script tests the decoupled architecture using Luxir, asserting schema auto-provisioning, full-text match queries, dense vector kNN search, OIS tombstone deletions, and Secure MCP Server endpoints.

• OpenTelemetry & Observability Pipeline:

  ./scripts/test-observability.sh

  This script tests the telemetry ingestion pipeline, spinning up Jaeger, the OTel Collector, and Prometheus, executing a JUnit-based trace emitter, and verifying trace propagation using Jaeger's query API.

• Secure Model Context Protocol (MCP) Server Decoupled Pipeline:

  ./scripts/test-mcp-decoupled.sh

  This script tests the entire Secure MCP Server capabilities in decoupled deployment mode, validating Streamable HTTP transport handshakes (JSON-RPC 2.0 initialize, session negotiation, ping), tool discovery (tools/list), AIOps observability tools (fetchJobTraces, getErrorLogs, queryThroughputMetrics), Zero-Trust ACL evaluation (secureVectorSearch) with explicit deny precedence, document content retrieval (getDocumentContent), accessible sources cataloging (listAccessibleSources), multi-dimension querying, and OIS tombstone deletions.

• OpenFGA Document Authorization Decoupled Pipeline:

  ./scripts/test-openfga-decoupled.sh

  This script runs the decoupled stack with OpenFGA (oc-mcp-server in openfga mode) and verifies that permissions ingested through the OIS endpoint are mirrored into OpenFGA, that MCP secure search returns to each caller exactly what OpenFGA allows, that a group membership written to OpenFGA grants and revokes access without re-indexing, that permission changes and tombstones move the tuples, that search fails closed while OpenFGA is down, and that the oc.authz.* metrics are published. See Document Authorization.

• Hyland OpenArch Integration Test Suite:

  ./scripts/test-openarch-integration.sh

  This script verifies bidirectional OpenArch harvesting, Parquet vector bridging conforming to embedding.avsc, Zero-Trust ACL preservation, and MCP search/fetch backend tools.

• Hyland OpenArch Decoupled Pipeline:

  ./scripts/test-openarch-decoupled.sh

  This script tests the entire decoupled architecture with Hyland OpenArch, validating document ingestion, Parquet vector bridging conforming to embedding.avsc, Kafka consumers (oc-crawler, oc-ingestion-consumer, oc-embedding-consumer, oc-writer-consumer), paged HXQL harvesting with Java 25 Virtual Threads, Secure MCP Server endpoints (openarch_search, openarch_fetch_document), and OIS tombstone deletions.

• OpenCrawling CLI (oc) Integration Test Suites:

  # Local CLI Integration Suite
  ./scripts/test-cli.sh
  
  # Decoupled Multi-Container CLI Integration Suite
  ./scripts/test-cli-decoupled.sh

  These scripts compile oc-cli, execute end-to-end command suite assertions (job creation, connector checks, system metrics, AIOps traces, schema validation, and archetype scaffolding), and verify live CLI interoperability against decoupled multi-container clusters.

---

Option B: Run OpenCrawling Locally (Development Mode)

If you wish to run the JVM runtime and React frontend directly on your host machine for development:

1. Build the Project (Maven)

Compile all modules using Java 25. Since we utilize advanced features, preview features must be enabled:

mvn clean install

2. Run the Ingestion Runtime Bootstrap

Start the Spring Boot runtime application:

mvn spring-boot:run -pl oc-runtime -Dspring-boot.run.profiles=dev

3. Run the Embedding Service Microservice

Start the Embedding Service application in a separate terminal:

mvn spring-boot:run -pl oc-embedding-service

Running a Sample Ingestion Job on Startup (Optional)

By default, the automatic startup crawl is disabled to prevent unnecessary scans. To trigger a demo crawl job on startup, pass the configuration properties:

mvn spring-boot:run -pl oc-runtime -Dspring-boot.run.profiles=dev \
  -Dspring-boot.run.arguments="--spring.opencrawling.crawl-on-startup=true --spring.opencrawling.scan-path=/your/local/directory/to/scan"

4. Run the Admin UI

To launch the administration dashboard:

cd oc-admin-ui
npm install
npm run dev

Open http://localhost:5173 in your browser.

---

Scaling Out & Performance

OpenCrawling is designed for high-throughput, horizontal scalability. Since the ingestion pipeline is decoupled using Apache Kafka and the Claim Check Pattern, you can scale components independently.

1. Scaling the Ingestion / Processing (Output Connector)

Vector indexing and embedding generation is typically the primary performance bottleneck because of deep learning model inference (Ollama/OpenAI) and database indexing (pgvector).

• Kafka Consumer Group Partitioning: The three main topics (opencrawling-ingestion, opencrawling-chunks, and opencrawling-embedded) are consumed by IngestionConsumer, EmbeddingConsumer (in oc-embedding-service), and VectorStoreWriterConsumer respectively within the OpenCrawling services. By configuring these topics with multiple partitions, Kafka distributes load dynamically among active consumer nodes.
• Horizontal Scaling of Service Instances: You can run multiple instances of the oc-embedding-service application sharing the same consumer group. Kafka automatically distributes partitions and load-balances the messages.
• Ollama Load Balancing: Scale out embedding generation by pointing baseUrl to a load balancer (e.g., NGINX, HAProxy) backed by a cluster of Ollama instances running on GPU-enabled nodes.

2. Scaling the Repository Connectors (Ingestion Source)

The scanning/crawling phase can be distributed by splitting large target sources:

• Partitioned Scans: Run separate bootstrap crawl jobs targeting different sub-directories or repository prefixes.
• Distributed File Shares / Shared Storage: In a multi-node setup, ensure the IngestionConsumer instances have access to the same shared filesystem (e.g., NFS, S3/MinIO bucket, SMB) as the repository crawlers, so the Claim Check reference (path/URI) can be successfully resolved by the consumer node.

3. Claim Check Pattern

To ensure the messaging system remains fast and responsive:

1. The Repository Connector crawls data, but instead of publishing the entire document content (which could be megabytes of binary data) to Kafka, it saves/references the file on a shared storage medium.
2. It publishes a lightweight IngestionMessage (Claim Check record) to the Kafka topic containing the metadata (URI, file path, version).
3. The Consumer Workers process the ingestion:
   • IngestionConsumer pulls the reference, reads the file directly from storage, extracts text with Apache Tika, splits it into semantic chunks, and publishes them to the chunks topic.
   • EmbeddingConsumer (running in the oc-embedding-service microservice) pulls the chunks, reads the dynamically configured Transformation Connector engine configurations, requests embedding vectors from the target model engine (Ollama, OpenAI, Hugging Face, etc.), and publishes the embedded chunks to the embedded topic.
   • VectorStoreWriterConsumer consumes embedded chunks and uses a stateless PrecomputedEmbeddingModel to save them directly to pgvector.

---

Verification & Monitoring

• Database: Access PostgreSQL at localhost:5432 (User: opencrawling, DB: opencrawling).
• Redis Dashboard: Open http://localhost:8001 in your browser to view the Redis Stack Insight dashboard.
• Logs: Monitor console output for the Virtual Thread Executor and Structured Concurrency task logs.

---

Troubleshooting

• Java Version Check: Run java -version to confirm you are using Java 25.
• Preview Features: If your IDE fails to compile structured concurrency code, verify that the --enable-preview JVM argument is configured for compiler and runtime settings. (It is already pre-configured in pom.xml).

---

📬 Contact & Support

Join our Slack Community to chat with developers, share feedback, or ask configuration questions.

For general inquiries, community updates, or security-sensitive disclosures, please contact the maintainers at info@opencrawling.org.

---

Trademark

OpenCrawling® is a registered trademark of the OpenCrawling Organization. For guidelines on using the name and logo, please refer to the TRADEMARK.md file.